Scripts Part 3: Split and Rejoin Large Files

Two events have transpired to lead to the posting of this script: 1) I've been meaning to post a new script to the Scripts section of the blog for some time, and 2) MacOSXHints today had a hint about splitting and rejoining large files, and suggesting scripting this process. I just happen to have had such a script lying around for some time, so this seemed like an especially appropriate time to post it.

Essentially, this script is made to take large files and cut them into smaller (theoretically CD or DVD sized) pieces. The same script can then be used to rejoin these chunks. For splitting, the script uses the split command, and to rejoin files, it uses the cat command.

So here it is in all its glory.

SplitAndRejoinFiles Script
See the code

Scripts Part 2: .DS_Store Remover

Why do so many Mac folk need a GUI for everything? Sometimes a little script is plenty to get the job done. And using a script can be just as easy as using a GUI: it's double-clickable; it's instructive; and it's drag-n-drop. It just doesn't have pictures. But for some reason people still freak out when they see that big, scary, text-based Terminal pop up. I used to be that way. But I've grown to love the power of the command-line, and simple shell scripts.

So here's another popular favorite: The .DS_Store Remover utility. There seem to be billions upon billions of little apps that do nothing but remove the .DS_Store files that the Mac OS puts in every folder on your system. Maybe there are so many of these little apps because it's actually such an easy thing to script. The nice thing about the script is, you can see and modify the code very simply and easily, whereas with one of these GUI wrappers, God knows what it's actually doing.

For those who don't know, the .DS_Store file holds information about how a given folder should display when it's opened. It remembers the position, size and view style (i.e. column, list or icon) of the window. Why remove them, you may ask? Some people don't like them showing up on their windows boxes, apparently. Too messy. Ahh, if only I could write a script to delete Windows boxes. That seems like the better approach.

Anyway, here's my script version. I call it "DSStoreRemover," which frankly I think is possibly the most ingenious and creative name for anything ever in the history of humans on planet Earth.

It's a really, really, drop-dead stupid simple script that takes a folder path as a variable which is used in a find command that then, in turn, executes rm on anything named ".DS_Store."

See, you don't need a GUI for this. Hell, you almost don't need a script.

.DS_StoreRemover Script
See the code

Scripts Part 1: Maintenance

I see a lot of shareware applications out there, and some of them are great, but then some of them make me think how easy it would be to write a simple shell script to do the same thing they do. The primary advantages here are twofold:
1) The script, as opposed to the application, is free.
2) The script can be run from the command-line, and, therefore, remotely.

If either of these advantages are more a concern to you than a pretty wrapper around what is essentially a series of shell commands, then this series will be for you. I have been writing a lot of what I think are pretty useful little scripts. These scripts are stupid simple, and I really make them so I don't have to type out a bunch of commands to do certain things, and so that I don't have to scour man pages for syntax when I invariably forget the proper way to write out a command.

I am by no means an advanced scripter, and I'm sure folks can find lots wrong with my scripts, and lots of ways to do things better. And that's great. I really want to learn more -- and better -- scripting methods. So feel free to leave suggestions on the site in the comments section.

Here is the first of many featured scripts. It's a really simple idea inspired by one of the bajillon shareware dealies out there that does what they call "maintenance:" It runs daily, weekly, and periodic maintenance, it will update your system's prebinding, and it will repair permissions. This script does exactly what many of those apps (essentially, GUI wrappers) do, but instead of giving you buttons and checkboxes, you get to enter text in the shell, which, I promise, will make you feel a hundred times cooler. Well, in the geek sense of the word, anyway.

So here it is, for your pleasure. Look for more scripts on an irregular basis in the near future.

Multi-Maintenance Script
See the code

Why We Need Anti-Virus Software for Mac

I recently wrote a review of the excellent antivirus utility, ClamXav. I also read constant articles and hear constant debate about whether or not you need virus protection on the Mac. I used to be in the camp that says, "There are no viruses for Mac, so why use antivirus software?" But nowadays, I find myself in the other camp, the one that says, "Of course we need virus protection on the Mac, you idiot."

To be honest, I was never as cavalier as to suggest that no virus protection was ever needed on Macs. But we Mac folk are in an interesting predicament (though not as interesting as our Windows-using pals): Currently no viruses directly affect us, and antivirus software for Mac is, by and large, abhorrent. In fact, it is far more likely that your system will be adversly affected by antivirus software than it will by a virus. To wit, Norton Anti-Virus has frequently caused numerous problems on client Macintoshes I manage in my freelance duties. Moreover, many antivirus software packages install kernel extensions, which is the surest way to hose a system. Even Apple recommends against it to developers, citing kernel extensions as a last resort. I frankly don't understand why antivirus software would have need of kernel extensions, given that all it really needs to do is scan files and compare them against a list of known viruses, but apparently the Norton folks think this is important. And it's been wrecking people's systems.

So, the state of things being what they are, it's no surprise that Mac users just go, "Fuck this," and ignore the problem, or worse, deny it. I mean, what else is a poor Mac gal or fella to do?

Let me back up here and explain why I've switched camps. There are two reasons, actually. One, I work in a very heterogenous network, and I see the effect Windows viruses can have on our systems. And on our Windows admin. It's hellish. And it's a problem that, while I don't personally suffer from it, I certainly don't want to contribute to. Macs can and do spread viruses to other computers. I've seen it happen. At this point I could launch into a whole number about how we're all citizens of the internet, and how it's our responsibility to be good ones. But I won't. Instead I'll tell you my second reason for switching camps: I got a virus. Yep. Sure did. This virus (actually, I think it was a worm, but we'll treat all such programs as "viruses" for the purpose of this article) was passed to me, I believe, by a Windows user inside a Word document. Unfortunately, I needed this document, and I needed to send it back out to other Windows users. Fortunately, I had a trusty old copy of Norton Anti-Virus and an OS9-bootable system from which to do the repairs. But if I hadn't, I would not have been able to use the document. If my job had been dependent upon that document... Well, you can extrapolate. Unless you're planning on never sharing files with anyone other than Mac users -- ones who also only share files with other Mac users, by the way -- you do have to worry about viruses. Just not as much as Windows users. Here I like to paraphrase the AIDS prevention folks: When you're sharing files with someone, you're sharing files with everyone they've ever shared files with. And the internet is, like, one big, giant file-sharing orgy. Do you really want to be running around out there without a condom?

Me neither.

I don't want to get too much into the options. This is more an explaination of why we Mac kids do actually need some form of virus protection. But I will quickly tell you what I do, and why I've settled on my method. My method is the ounce of prevention method. I use ClamXav on my systems and do weekly scans. Also, using ClamXav's new "Sentry" feature, I have a few watch folders: my mail, my downloads folder, and any folder I might be sharing on my LAN. (Keep in mind here that ClamXav does not scan subfolders, for performance reasons.) This pretty much covers most of the bases. If you get ClamXav set up right, you should be in real good shape when it comes to detecting viruses. Unfortunately, ClamXav does not repair viruses. So if you already have one, or if, God forbid, one should squeak by, you'll need something to fix it. I'm lucky. I have my old OS9-Norton system. But these are becoming almost as rare as Mac viruses themselves. If you have a virus now, you should quarantine all instances of that puppy, go do some research, and find the least invasive, non-kernel extension installing antivirus repair software you can. If you can run it off the CD without installing anything, all the better. Otherwise, just wait. Yeah, you heard me. Wait. The chance that you'll get a virus is pretty slim, and it's quite likely that, by the time you do, any virus software you buy today will be out of date, obsolete, or just plain useless. So wait, and if a virus ever rears its ugly head on your system, then go buy something to fix it. Oh, I might also suggest that if the antivirus software does have to be installed on the system, you might want to use a spare firewire drive for the install, provided you have one, of course. I like to have a lean, bootable OSX system on a small firewire drive, install the antivirus software there, and boot from this drive when I have a problem. That keeps my primary boot drive clean of antivirus cruft.

So that's what I do. And that's what I think. And so far, it's worked pretty well. The only thing that kind of breaks my flow is when freelance clients freak out and install Norton AV on their systems without asking me about it first. Ever try to remove that shit? Holy Hell. Thank my lucky stars for this uninstall script, but until I found it, it was murder.

Okay, kids. Time to go put a helmet on that soldier.

Tiger Lab Migration Part 7: Wiggly-Niggly

Well, we're getting close. We're down to last, wiggly-niggly little tidbits. Here's where we've been in the last few weeks:

The Master System
We have completed a very nice build of Tiger with most of the software we need on all machines, though we're lacking all the most recent Adobe stuff due to not having received the most recent Adobe stuff. 'S'okay. We'll make do with CS1 for now. Our system is all loaded up with our custom scripts, and a new, beefier home account mount script. And it's got a nice, big, 75GB system partition. Why, it's just swell.

We've used this system to clone to all our other lab machines. We boot these other machines in firewire target disk mode, and then we use Disk Utility to restore the Master System to the new machine's system partition. Or, we wanted to use Disk Utility. Turned out that our first Disk Utility-cloned system had some problems due to -- yep, you betcha -- Tiger bugs. The problem was that Disk Utility wasn't blessing the new system partition, so when we'd boot the new machine, we'd get the flashing question mark of doom. So we'd have to then boot off the Tiger install disk, and set the new system partition on the newly built machine to be the startup disk. I decided this was a pain, and that it would be much easier and more instructional to do our clones with a tiny, tiny shell script that calls ASR (Apple Software Restore, via the asr command).

#!/bin/bash

## Script to Clone Local Volumes #### systemsboy - Aug 18, 2005 ##

clear

## Define Variables ##

echo "Please enter the path to the SOURCE volume or ASR-ready disk image...(Example: /Volumes/MyVolume)"read SOURCE

echo "Please enter the path to the TARGET volume...(Example: /Volumes/MyVolume)"read TARGET

## ASR Portion of Script ##echo "Building Clone... Please wait...________________________________"sudo asr -source "$SOURCE" -target "$TARGET" -erase

exit 0

This did the trick indeed, and was as easy to use, if not as purdy, as the Disk Utility application. This method only yielded one problem, and that was that all the invisible folders -- /var /etc /private and the like -- were not made invisible on our cloned systems. This is another Tiger bug, but one that was fixed in 10.4.2's version of Disk Utility. Apparently not fixed in ASR from the command-line, however. This problem yielded both an intersting solution, and some interesting information. Back in the day, these files were made invisible to the Finder by the magic of the .hidden file. The .hidden file was simply a list of files or folders at the top level of the root drive that should be treated as invisible by the Finder. Want to make something invisible on /? Just add it to the .hidden file and restart the Finder. (Ironically, the .hidden file was also at the root level of the system, but was rendered invisible by the fact that its name began with a period.) Nowadays, however -- in Tiger, that is -- this is no longer that case. Files on root are now made invisible, according to this Apple KBase article, by setting certain file attributes. I'll tell you, I have yet to figure out what those attributes are, or how to modify them, but when I do I'll shout. In the interim, I was forced to locate the "SetHidden" program on my original Tiger install DVD. After insering the DVD, it can be found at:

/Volumes/Mac\ OS\ X\ Install\ DVD/System/Installation/Packages/OSInstall.mpkg/Contents/Resources/

I've now copied this file to a folder called SetFile, which I've placed in the Applications folder for future use. Also, I needed the hidden_MacOS9, which can be found, and should be placed, in the same folder as the SetHidden command. So I now have a folder in my Applications folder, called SetHidden, in which the SetHidden command and the hidden_MacOS9 files reside, installed on my master, and I've added the commands:

cd /Applications/SetHiddensudo SetHidden /Path/To/My/Clone/System hidden_MacOS9

to my shell script, and voila! Clone script complete!

#!/bin/bash

## Script to Clone Local Volumes #### systemsboy - Aug 18, 2005 ##

clear

## Define Variables ##

echo "Please enter the path to the SOURCE volume or ASR-ready disk image...(Example: /Volumes/MyVolume)"read SOURCE

echo "Please enter the path to the TARGET volume...(Example: /Volumes/MyVolume)"read TARGET

## ASR Portion of Script ##echo "Building Clone... Please wait...________________________________"sudo asr -source "$SOURCE" -target "$TARGET" -erase

## Set Hidden Files (Tiger Bug Workaround) ##echo "Mac OS X 10.4 does not properly set hidden files.We will now use the SetHidden command to rectify the problem.For this step, the SetHidden folder must be present in /Applications.---------------------------------"cd /Applications/SetHiddensudo ./SetHidden "$TARGET" hidden_MacOS9

exit 0

Cloning all 14 systems on the floor took my Lab Assistants a few short hours.

Spotlight
I mentioned some concerns regarding Spotlight and the particular way in which users use our lab. After some mucking around, and testing of various methods of disabling Spotlight, both forced and optional, I've decided to do nothing. I do this kind of thing all the time. Worry and worry about something, and then, finally, throw caution to the wind and do nothing about it. It's my way. It's part of what makes me so gosh-darn lovable. But I do have two things behind that decision: a rationale, and a backup plan.

The first problem I'd anticipated was that Spotlight would begin indexing user home accounts over the network all at once as soon as everyone logged in, and that this would cause problems with network performance and incomplete indexes. Upon further reflection (read: obsessing), I decided that that Spotlight's network resource usgae would probably be low, and that it would probably avoid incomplete indexes by either continuing in the background after logout, or by resuming at next login. Also, since home accounts have quotas, the largest of which is 7 GB, indexing shouldn't take too long, and any problems should be quickly mitigated.

The second possible problem I'd worried about was that Spotlight's insistance on indexing firewire drives would take forever to yield complete indexes on large drives -- a process that would likely be interrupted at logout before it was complete -- leaving many users with partially indexed drives. In my tests, however, Spotlight picked up right where it left of with drives that were partially indexed when ejected and then re-mounted, so I'm not convinced this will even be a problem. I also worried that Spotlight's firewire drive indexing would cause performance problems that would incapacitate users who were working on video. This may yet be the case. We'll know soon enough.

I came up with a number of possible solutions to these problems, all messy, all imperfect. Ultimately, I decided that the best option was to wait and see what problems arose, and then tackle those problems with the most appropriate of the solutions I devised. Who knows? Maybe it will all work out great without any intervention from me. And that'd be just ducky.

Network
We have an intersting and complex network. It consists of Mac, Linux, and Windows machines, a network RAID, and web and mail storage. And all of this is accessible, in one way or another, via the network. The problem comes when you try to explain all this to new (or sometimes even old) students. It's not an easy picture to paint. And overwh elmed first years usually don't quite catch on for some time. But as we design our network, and build our systems, we're very concerned about how all that is presented to the users. And we try to make it as simple and as understandable as possible, which is, after all, the Mac way, and is why people love the Mac. So let me just say, right here, right now, what the fuck is up with the "Network" view? That piece of shit gets worse with every OS revision. And no one seems to care. I can't find diddly about it on the forums.

So here's the background on this outburst. Back in the Panther years, the Network view offered up a pretty nifty way to present a great deal of our network -- the shared drives of all platforms -- to our Mac users. The coolest part was that it looked much like what you saw on Windows, which simplified things greatly by presenting a consisten network view across platforms. There were a couple things that made this possible. One was SMB workgroup names. SMB does a great job of not only sharing files, but broadcasting itself as a service on a network. Setting up computers under SMB with a workgroup name based on the platform of the given machine got all our Mac, Linux and Windows computers grouped appropriately on the network. And Mac can read those groups and smartly creates folders in the Network directory named after those groups and populated with the systems that belong to them. So in my Network browser, there'd be a folder called "Linux," and all the Linux computers would be in there. Same for Windows. Here's the catch: Our Macs share via SMB to Windows, but we want them to share via AFP to other Macs. Personal File Sharing, which uses AFP, does not, by default broadcast any kind of grouping. If you turn on AFP, you just broadcast the service, but you can't specify it's appearance or where on the client the shares will appear. Computers broadcast via AFP in Panther just show up in the "local" folder, not in a "Macs" folder like their SMB sharing counterparts. There was, however, a sneaky way to define this in Panther, by editing a little file called /etc/slpsa.conf, and thereby magically creating what are known as SLP scopes. In fact, SLP scopes are just that -- a way to group a particular service from a particualr device into logical groups. So we scoped our Macs to the "Macs" scope and they showed up in our "Macs" group in /Network, and all was well in the universe. If you're unbearably astute, you'll probably be wondering why, though, our Macs, when connecting to other Macs via the "Macs" scope in the Network folder, didn't connect to said Macs via SMB rather than AFP. And here was something amazingly cool in Panther. Panther smartly dealt with the folder full of Macs in our Network view. So, when it saw these other Macs broadcasting and sharing over two protocols -- AFP and SMB -- it intellegently chose the best one, which for a Mac was AFP, of course. That was the thing of beauty that allowed us to present the same view of the entire, heterogeneous network the same way on all platforms while still allowing us to connect with different protocols. It was frickin', frackin', goddamn sweet, is what it was. Brilliant. And now it's gone.

Not only is this intelligent decision making gone from Tiger, at least in the latest incarnation (10.4.2 as of this writing), but so too is the ability to define SLP scopes. So, this means two bad things: 1) I can no longer define specific locations for my AFP-shared Macs, and 2) any connection made via the "Macs" folder in the Network folder (which, remember, is now only defined by SMB, not by SLP) connects via SMB; my Macs connect to each other via the Windows file sharing protocol, which is a big fat problem.

So, I'm currently trying to devise a way to scope the Mac shares into some sort of logical grouping that will allow me to present those shares in a palatable way. But it ain't easy. I know you can do this with Tiger Server, and this is probably why they've crippled it on the client, but I don't a Tiger Server yet, and I'm hesitant to build one this close to the beginning of the semester. So we shall see...

The migration is nearly complete. Time permitting, I would abasolutely love to build a fresh new Tiger Server. I'll probably give it a shot in this last week before classes begin, but I'll be sure and keep a clone of the existing Panther Server in case things go awry. Beyond that, and the aforementioned wiggly-niggly, we're done. We're upgraded, on the workstations anyway. And it is good.

There will, most likely, be problems that arise once the students come in and start using the new systems. I will share those in a later post. Until then...