Comments on: Find Gremlins With BBEdit

By: dave e

dave e — Thu, 28 Feb 2008 07:46:25 +0000

this whole thing just bit me in my scripting. I now manually check for these but now that I can automate it makes it even better.

thanks daniel for the great support !

Dave in Anchorage

By: Charles

Charles — Wed, 27 Feb 2008 10:11:23 +0000

This has bitten me a few times too (more often when creating HTML from TextEdit using docs people have created on Windows). Tex-Edit Plus, the still-going free text editor, has a “cleanup” function which will strip out high ASCII (which these usually are to cause the problem; low ASCII such as CR/LF etc don’t trouble it).

But I’d echo other folk saying that it would be good to have highlighting, at least for high ASCII (red?) and maybe even for low ASCII (green?). Though as you say it might turn out to be one of those functions that only 0% really need, and those who don’t know about it would still send support queries. Unless, hmm, you had such a function turned on by default – people would see the measles spots in their text and wonder what it was.

By: Cameron Hayne

Cameron Hayne — Wed, 27 Feb 2008 08:47:27 +0000

In case someone finds it useful, here’s a Perl script that checks for “gremlins” in text files and prints the lines containing them.
(I’m using the PRE and CODE tags around this script in the hope that it might make it more readable – it doesn’t seem to work in the live preview but I’m hoping that is just a bug.)

#!/usr/bin/perl


# announceGremlins

# This script is intended for checking text files for the presence of gremlins.

# (A "gremlin" is the BBEdit name for an unwanted character, often invisible,

#  that has somehow gotten into your text file.)

# The current implementation of this script looks for characters that

# are not in the POSIX printable character class.

# You can supply filenames as command-line arguments, or send the text you

# want checked via STDIN.

# Cameron Hayne (macdev@hayne.net)  February 2008
use strict;

use warnings;

while () { chomp; my $line = $_; while ($line =~ /[[:^print:]]/g) { my $colNum = pos($line); # we want a 1-based column number my $before = $`; my $after = $'; print "Non-printable character at column $colNum of line $. of file $ARGV\n"; print "$before?$after\n"; } } continue { # reset line numbering on each input file close ARGV if eof; # Not eof()! (eof with parentheses is different) }

By: Miraz Jordan

Miraz Jordan — Wed, 27 Feb 2008 00:09:29 +0000

I’m one of that 0%. These things just creep in from copy and pasting, or from other sources, by mysterious means.

I have BBEdit and often use Command J to edit posts there before sending to my blog. The red upside down question mark is very handy in BBEdit.

It would be kind of useful to be allowed to display invisible characters in Mars Edit, for those of us who do lots of writing, and editing quotes from others….

By: sairuh

sairuh — Tue, 26 Feb 2008 18:19:38 +0000

I ran into this recently after pasting stuff I had copied from a log file (from Console) into a blog entry. Was I glad to have found a solution in the MarsEdit forum to fix the problem!

Daniel, that’s a great idea to highlight or display warnings where these wacky characters appear in the content.

By: Daniel Jalkut

Daniel Jalkut — Tue, 26 Feb 2008 17:52:02 +0000

Gruber: I’m not 100% sure, but I think the fact that the invisible characters are still legal “ASCII” – they’re a legal part of UTF8.

By: John Gruber

John Gruber — Tue, 26 Feb 2008 17:46:17 +0000

The reason these gremlins render the resulting XML invalid (I think) is that they’re *bytes*. Unicode can support any character, but you can’t insert random bytes in the middle of a UTF-8 stream. BBEdit’s Show Invisibles command is useful for spotting these things, too.

By: Daniel Jalkut

Daniel Jalkut — Tue, 26 Feb 2008 16:14:44 +0000

Mike: That’s a good question. I suspect they’re never useful, but one of the things I am sensitive to with MarsEdit is not “mucking with” the blogger’s content. So I’d want to be very sure before automatically stripping them out. Currently what I’m thinking is maybe I can just make them show up as very visible red warning icons in the text. Then you’d be able to see it and delete it, but also leave it if it’s for some reason purposeful.

Ölbaum: I don’t think the XML being generated is technically invalid, but just triggers a problem on the servers it sends to (in this example, WordPress). But that raises a good question: perhaps this should also be reported as a bug to systems that don’t handle such content gracefully.

Daniel

By: Ölbaum

Ölbaum — Tue, 26 Feb 2008 09:30:28 +0000

I have a hard time understanding how an invisible character can render the XML of the request not well-formed. XML uses Unicode. If the ‘gremlin’ comes from the body of the post, then it should not be a problem. The only way I could see a problem is if they appear in the XML tags, which should not happen as the tags of the request are generated by MarsEdit and not directly from user input.

By: Julian Grey

Julian Grey — Tue, 26 Feb 2008 08:48:56 +0000

I usually end up with these when cutting and pasting text from a web page open in Safari to BBedit, the funny thing is that it never happens when cutting from Firefox.

Hmm but I have never ended up “keyboarding” them in. I would be nice to figure out where they come from.