00667nam _22002414_245__002000800000005001900008008004300027020001500070035001000085039001800095040002300113082001600136245005700152250001100209260004300220300002400263500002000287500003400307852002600341100002100367650002000388650001700408#$a13568#$_00000000000000.0#$_030129s1992____--------------000-y-eng-d#__$a0314922180#__$a13568#80$a2$b1$c_$d_$e1#__$aZA$aF101$c20030129#04$220$a621.395#10$aFundamentals of logic design /$cCharles H. Roth, Jr.#__$a4th ed#__$aSt. Paul, MN :$bWest Publishing,$c1992#__$axviii, 770 p :$bill#__$aIncludes index.#__$aIncludes answers to problems.#__$aF101$b_N$c621.395 ROT#$aRoth, Charles H._1#_0$aLogic circuits.#_0$aLogic design#*

I used the fact that both Sublime Text and Notepad++ support find and replace using ‘regular expressions’. Regular expressions are a powerful way of matching text strings. As well as simply making a precise match to a word or piece of text, you can use Regular Expressions to match types of characters (e.g. ‘match a digit’, ‘match a lowercase letter’), or sequences of characters.

For example, you can see in the example above that there is a repeated pattern of a ‘#’ followed by two characters, followed by ‘$’ – this is the record separator (start of a new MARC field) followed by two indicators, followed by the unit separator – the first subfield in the MARC field. The regular expression that will match this is:

#..\$

The ‘#’ matches the # character – just a simple match. The ‘.’ (period/fullstop) character is a wildcard that will match any other character – so the two dots mean ‘match any two characters’. Finally the ‘\$’ matches the dollar sign – the ‘\’ in front of it is needed because a ‘$’ on its own in a regular expression has a special meaning of ‘end of the line’ – the ‘\’ forces it to match the dollar sign directly (using a ‘\’ in this way is called ‘escaping’ the special character).

I was pretty confident that this pattern wouldn’t appear randomly elsewhere in the file – so I could use this as a first find/replace. I wanted to replace the # and $ with the record separator and unit separator characters, but keep whatever two characters were between them. You can do this using regular expressions with something called a ‘capture group’ – where you put the characters you want to ‘capture’ (to be able to use in the ‘replace’ expression) using brackets ‘(‘ and ‘)’. So the Find expression becomes:

#(..)\$

You can capture multiple groups in a single Find expression, and these are just numbered from 1. The syntax for using the ‘captured’ group in the Replace command varies – in Sublime Text it is ‘\1’, while in Notepad++ it is ‘$1’ – but they do the same thing – take whatever value was captured, and use it in the replace expression.

Using this approach I was able to replace the ‘#’ and ‘$’ with the appropriate ASCII characters (which I copied and pasted from a valid MARC file I already had – these special characters didn’t actually show visibly in the Notepad++ Find/Replace dialogue – but they were there and do work in a replace statement).

Once I’d done this I was also able to replace the ‘#*’ that appeared at the end of every line with a Unit Separator followed by a Group Separator. I checked that this replace action replaced the expected number of characters to reassure myself that this had worked OK.

Working with 50,000 lines in a text editor like Sublime Text or Notepad++ can cause some problems. This kind of text editor typically holds the whole of the file in active memory while you are editing the file – and large files can take up large amounts of memory. Fifty thousand lines did make Sublime Text and Notepad++ slightly sluggish in their response times, but happily it didn’t prove too much for either of them.

If I’d hit problems with file size, I’d probably have tried using a tool called ‘sed’ instead. ‘sed’ is a ‘streaming editor’ (hence ‘sed’) which means it doesn’t try to hold the whole file in memory, and instead will run through the file bit by bit when you carry out a command. I have to admit that I don’t find sed very intuitive and often struggle to get it to do what I want. There is a good basic introduction to sed on Mac at http://www.maclife.com/article/columns/terminal_101_find_and_replace_using_sed, and sed is also available for Unix/Linux (natch) and Windows (a simple intro available at http://www.thoughtasylum.com/blog/2011/9/30/using-sed-on-windows.html)

\$[^a-z]

\$[^a-z2468]