I back up several Windows 2003 servers with BackupPC and rsync over ssh.  This is basically how I do it.

• Download cygwin 1.7+ from the regular place
• Install the default system, plus the following: cygrunsrv, openssh, and rsync
• I also install joe (my editor of choice), and procps (for top)
• Start a cygwin shell and type:

# ssh-host-config -y
# cygrunsrv -S sshd

• 5. I then attempt to ssh into my backupPC system from cygwin, just a nice test, and creates the .ssh directory for me.
• 6. Copy BackupPC’s public key over

# scp root@backuppc:/var/lib/backuppc/BackupPCkey.pub ~/.ssh/authorized_keys

• Install the following scripts to Administrators home directory

pre-backuppc.sh

#!/bin/bash
# script to create shadow copies of Windows drives and export them to a
# drive letter for BackupPC backup
# the shadow copies get mount to c:\shadow\(drive letter).  The directory
# structure must exist

# Launches passed input via 'at' to get around $USERNAME=SYSTEM
# problem under ssh login where the shell lacks permsisions to run
# commmands like vshadow or dosdev
function at_launch ()
{
    local h m s wait1 command
    if [ $3 != "" ] ; then
        command="${1} ${2} >> ${3}"
    else
        command="${1} ${2}"
    fi

    set -- $(date +"%H %M %S")
        h=$((10#$1))  #Note explicitly use base 10 so that 08 and 09 not interpreted as bad octal
    	m=$((10#$2 +1)) #Advance minutes by 1
    	s=$((10#$3))
    	wait1=$((60 - $s))
    [ $s -gt 55 ] && let "m += 1" "wait1 += 60" # Make sure >5 seconds left
    [ $m -ge 60 ] && let "m %= 60" "h += 1" #Overflow minutes
    let "h %= 24"
       	at $h:$m $(cygpath -w $(which bash.exe)) -c \"$command\"
#       	> /dev/null

    echo Running \'$command\' at $h:$m
    return $wait1
}

# create the command to shadow the drive, and wait 2 minutes plus the seconds
# before the at command runs before returning (make sure shadow copy is made)
function shadow_drive ()
{
    date
    echo Shadowing Drive $@
    local wait1
    drive=$@

    at_launch "/cygdrive/c/WINDOWS/vshadow.exe -p" $drive "/home/Administrator/vshadow-out"
    wait1=$?
    let "wait1 += 120"
    echo sleeping for $wait1
    sleep $wait1
    date
    echo done sleep
}

# get the guids from the vshow-out file and place them into the shadow-guids
# file for the post-backuppc scripts use (and hours for mapping to directory)
function get_guids ()
{
    echo Getting shadow copy GUIDS
    cat ~/vshadow-out | grep "* SNAPSHOT ID" | awk '{print $5}' >> ~/shadow-guids
}

function map_shadow ()
{
    echo Mapping GUID $1 to $2
    local wait1
    local guid="$1"
    local dir="$2"
    at_launch /cygdrive/c/WINDOWS/vshadow.exe -el=$guid,$dir "/home/Administrator/map.out"
    wait1=$?
    let "wait1 +=30"
    sleep $wait1
}

date

# get rid of the guids file if it exists
rm ~/shadow-guids
rm ~/vshadow-out
rm ~/map.out

sleep 10

# create the snapshots
shadow_drive c:
shadow_drive h:
shadow_drive j:

# get the guids into a single file
get_guids

loop=0

# create the shadow directory structure AFTER we make the shadow copies
# the post-backuppc.sh script deletes this tree after removing the mounts
mkdir /cygdrive/c/shadow
mkdir /cygdrive/c/shadow/c
mkdir /cygdrive/c/shadow/h
mkdir /cygdrive/c/shadow/j

# loop throuh the guids and map to mount point
# assumes guids in file are in order of shadows created
while read line ;
do
   if [ $loop == 0 ] ; then
       map_shadow $line "c:\\\\\shadow\\\\\c"
   fi

   if [ $loop == 1 ] ; then
       map_shadow $line "c:\\\\\shadow\\\\\h"
   fi

   if [ $loop == 2 ] ; then
       map_shadow $line "c:\\\\\shadow\\\\\j"
   fi

   let "loop += 1"
done < ~/shadow-guids

post_backuppc.sh

#!/bin/bash

# script to delete the shadow copies used by backuppc

while read line ;
do
  vshadow -ds=$line
done < ~/shadow-guids

# now clean up the directory structure
#rmdir /cygdrive/c/shadow

8. In the BackupPC config for the system (I use the web page to edit the config) Add the following

DumpPreUserCmd: $sshPath -c blowfish -q -x -l Administrator swtwd01.norscan.com /usr/bin/bash -l -c /home/Administrator/pre-backuppc.sh

DumpPostUserCmd:$sshPath -c blowfish -q -x -l Administrator swtwd01.norscan.com /usr/bin/bash -l -c /home/Administrator/post-backuppc.sh

* sometime between the release of Cygwin 1.7 and the current version, ssh login became case sensitive.  I used to log in with ‘-l administrator’ and now I have to log in with ‘-l Administrator’.  That one set me back a bit.

Once you do a change to the backups, make sure you do a full backup right away.  It’s a good test, and backuppc doesn’t like it’s backup directory structure changed during incrementals.

I’m now looking into the information found here: http://www.goodjobsucking.com/?p=219 to see if I can do a psuedo bare-metal restore on the servers.

And that’s it. I have BackupPC backup /cygdrive/c/shadow, and all is well. Now, if someone can tell me how to color/syntax highlight bash code in WordPress, I’d be happy.

And the end result — well — wow.  Boy did I get a cross-section.  Two readers never finished.  They both got about 40 pages in (more on that later) and called it ‘first draft material’ and ‘needs a lot of work’.  Neither are published (which isn’t that big a deal), but more importantly, neither read in the genre.  Reading that back, it makes it sound like genre writing is somehow ‘less’ or ‘lower quality’ that other writing.  That’s not what I’m saying at all.  What I am saying is that there are certain tropes that don’t need to be explained in detail if you are already familiar with the genre.

Two other readers greatly enjoyed the book.  One to the point of forgetting she was supposed to critique.  That makes a writer feel good.  The other almost cried at the end.  The ‘almost’ kinda got me.  I’ll need to work harder to get that from an almost to a definitely.

My fifth and final critiquer (sp?) also never reads in the genre, but she approached the critique from more of a line-edit/consistency point of view.  Very helpful!

The end result is that I need to strengthen the opening, add a bit of detail to make a stronger image of the city in the reader’s mind, and tweak the ending and a few conflicts in the novel, so that the ending has a more emotional impact.

It sounds like a lot, but it’s really not.  Just tweaks here and there.  Sheila Gilbert from DAW has asked for a full, so once I get these changes done, and have two new readers look at it, it’s off.

Now, back to my previous point of two readers only making it to page 40…  Before I send out my work, I ALWAYS spell check and re-read it.  I want to make sure things are as good as I can get them.  I don’t know what happened, but the document I checked is nowhere to be found.  I didn’t discover that until some of the beta readers came back to me with comments.  None of my spell check corrections were in there, and page 40 started a scene that was a duplicate of one on page 200 or so.  The thing is, it belonged on page 200.  The readers read it on page 40, and became completely lost.  And for good reason.  That scene needed the previous 200 pages in order to make sense.  It was embarrassing, to say the least.  And I still can’t find my good copy.

Now back to work.

Canvention (the Prix Aurora Awards) was part of Keycon this year, which usually means a fabulous writers track.  Things went a little wrong, and many of the visiting writers ended up with no panels.  I can’t complain though, because I wouldn’t be able to do the programming, so kudos to all those that try — they are better than I.

Robert J. Sawyer won the Best Novel Aurora for Wake, beating out the 4 other nominees.  I’d read 3 of the 5 nominees this year, so I felt good with voting this year.  Sorry Rob, if you read this, I voted for Hayden Trenholm.  At the rate Hayden is going, I’m sure his third (and final?) book in the series (Stealing Home) will be nominated next year.

The awards banquet was well done.  Thankfully they ran out of desert before they got to Sherry, Adria, (and here) and I.  We all had a fruit plate instead, and were the envy of all.

Sheila Gilbert , editor at DAW was there.  A pitch session was scheduled, which was great.  It wasn’t incredibly well done, those of us waiting to do pitches could hear the people making pitches.  Not great.  We tried to keep our conversations a bit loud, to create a bit of privacy for those pitching.  My pitch went okay.  Sheila asked for the full, but I have a feeling she asked for a full from anyone that pitched something that was completed.

Conventions usually drain me, and I end up sick for couple of days after the con.  This time, I went in with a sinus cold.  I was patient ’0′.  If you got sick at Keycon, it’s my fault.

Along with the regular panels, I managed breakfast with Hayden and his lovely wife Liz, a ‘before banquet’ drink with Rob Sawyer, Virginia O’Dine of Bundoran Press, Edward Willet, Hayden Trenholm and Liz, Sherry, Adria, a couple more I can’t recall right now.

Overall, a great weekend.  I had fun.

The one thing I truly got from that system, was a deep love for WordStar from MicroPro.  I took to the command sequences like a fish to water.  Even after the BigBoard died, I stuck with WordStar and it’s command sequences.  Turbo Pascal used the WordStar keys, one of the text editors I use today has a WordStar mode (joe).

I’ve even remapped my Caps Lock key to be a Ctrl key to make keyboard navigation easier. That being said, I do seem to have gotten into the habit of using the cursor keys for navigation.  Hmmm.

I missed my WordStar.  Oh, I always had a DOS or CP/M Emulator available that would run WordStar, but then I couldn’t print quite right.  The files couldn’t be read by other apps for printing.  It was generally a pain.

So, I decided to double my pain… I wrote a WordStar ‘clone’.  It’s got most of the command sequences in it, it’s missing most of the dot commands, and has only rudimentary formatting support.  Still, it’s getting there.  I can pretty much read and write WordStar 4 and under files.  Wordstar 5 and up is coming.

My next steps from here (once I get out of novel revision mode) is to

1. complete printing
2. complete the command keys
3. complete the dot commands.
4. add RTF read/write abilities (write for sure).
5. read and write all WordStar files

Then I’ll look into adding macros.  I’ve never used WordStar macros, so it’s a new one on me.

WordTsar (yeah, bad name) isn’t a WYSIWYG wordprocessor, but it does try make things look close.  For example, the screen width is the printer width (hard coded to 8.5 x 11 for now).  The screen shot above uses Times New Roman as it’s font, so variable width fonts are displayed correctly.  Bold, italics, etc are displayed as is as well.  The main code base can deal with font changes and display the correct font, but a user can’t change the font yet.  These font styles do not yet follow the WordStar font table.  I’m not sure if they will.  I’ve added a full screen mode, since I hate distractions when I write.  The program is UTF-8 throughout, except reading and writing of WordStar files, that’s still 8 bit ASCII.

It’s currently at version 0.0.1 Alpha, but it’s pretty stable and usable.  It has one crashing bug that I’m working on (weird delete problem).

I use wxWidgets to code it, so it’s cross platform: Windows, Linux, and OSX.  I currently only have the Linux version running, but next month, we’ll see.

Ahh, what I do to keep my programming skills up while I write!  Fun fun fun.

Doing revision is tough for me.  I prefer the feel of getting the first draft done.  Taking my outline and getting all of the raw thoughts onto the page.  Pieces of that first draft can be beautiful.  Pieces of it can also be the hideous stepchild of the devil himself.  That’s where revisions come in.

On my previous 3 novels, revision was the boring sludge I had to trudge through.  I hated it.  I didn’t want to do it.  Why couldn’t I be on a sunny beach on a tropical island instead.  Heck, -55° C with windchill was better than doing revisions.  Novel 4 was quickly turning in to the same slog.

I figured this was getting pretty stupid.  The work had to be done, so just f*cking do it.  That didn’t work too well.  What the heck was wrong?  I’ve been a programmer for much of my working career, and finding/fixing bugs (revising) was part of that.  Albeit with more immediate feedback on success or failure.  What caught my attention though was the pattern of work.  I’d already taken the programming process of requirements/architecture/design into my writing (all wrapped up in my outlines), so why not the bug fixing portion as well?  The only problem was that when something doesn’t work in a program, you can make changes and immediately see if you’ve fixed the problem or not.  In writing, whether the ‘problem’ is ‘fixed’ can be a matter of interpretation by the reader.

The end result is my current revision process, which seems to be working quite well… as long as I stick to the adage of ‘butt in chair’.  Still, getting my butt into the chair has become easier, and at times enjoyable, using this process.  Below are two sample pages.  The first one is heavily revised (though not the worst example I had) and the second is lightly revised.  The first page was roughly an hour and a half worth of work.

So what’s my process?  As you can see from the image above, the first part is that I do my revisions on paper.  I print out one chapter at a time, and then revise one scene at a time.  But I’m getting ahead of myself.

Step 1: After I’ve completed the first draft and done some plot/storyline and major blooper fixes, I send the novel out to my Alpha readers.  Alpha readers know they are getting what is essentially a first draft.  Their job is to comment on the plot, the characters development, the pacing, does it all tie together, etc.  Basically the big pictures items.  To me, this is analogous to a peer review of the architecture/design on software.

Step 2: Once I get back the reviews, I sit down and look at all the comments.  Do I think they are right?  Should I make some changes?  What kind of changes?  All this is done in my head for the entire novel (with some note taking) before I sit and do any work.

Step 3: Sit and do some of the bigger stuff on the computer.  The big change things I missed like hair color changing, different names, timelines problems if time is a factor in the story, etc.

Step 4: Print out a chapter, starting with chapter 1 and moving sequentially through the story.  Take the first scene and look at the notes from the Alpha readers.  Are there any specific comments I want to look at for this scene?  If so, I’ll do that first, revising in pen as necessary.  Once that’s done, I read the entire scene out loud.  It gives me a sense of what the scene is about, the pacing, and the flow.

Step 5: I look at the scene and see if it makes sense in the overall story.   Does the story advance because of the scene?  This is a big yes or no question.  If the answer is no, I put a line through the entire page (all pages for the scene).  Then I double check what was cut when I did that.  Is there anything I want to carry over to another scene?  If there is, I write it down in a notebook and continue to the next scene.

Step 6: Okay, I’m keeping the scene.  Is the scene a story in miniature?  Does it have a beginning, middle, and end?  Is there conflict or tension?  Is the conflict resolved, for good or for bad?  If it’s missing any of these, I re-arrange the scene so that I have them.  I’ll move paragraphs, lines, whatever is needed.  All in pen on the paper.

Step 6a: Does this scene reference anything I cut out earlier?  If so, toss it or figure out a way to get the information in another way.

Step 7: Now I look for specific components in the scene.  Is there dialogue?  If not, why not?  Would the scene be better with dialogue?  I go through the same process for engaging the 5 senses.  For a single scene, engaging all 5 senses can be too much for the reader.  I’ll usually pick 1 to 3 senses and make sure I’ve hit them hard enough to make the scene feel real.  I do this a paragraph or line at a time.

Step 8: Description.  When doing first drafts, my descriptions are pretty weak.  The car was black, or Her hair was wet. Sometimes, thats good enough.  Sometimes you want more, you want that kick.  The cars black exterior sucked the light right out the air, throwing everything around into twilight shades of blue.  Her hair, still wet from the sudden downpour, dripped into her eyes, blurring her vision and making the car look feral. Can you kick it up like that every time?  Only if you want your book to be put down while the reader rolls his/her eyes.  But you still need it sometimes.   A good place to think about kicking it over the top is when the character has a reaction to external (or internal) stimuli.  If the stimuli warrants it, make the reaction visceral, and believable.

Step 9: Once I go through the whole scene, I go back to my notebook.  Is there anything I cut out earlier that I want to (or could) place into this scene.  Does the insertion work here?

Step 10: Repeat steps 7 – 9 for each paragraph.

Step 11: Read the entire scene out loud again, looking for the same things as in Step 4.

Step 12: Start at Step 4 for the next chapter/scene.

Step 12a: Sometimes I see (either by my reading or from Alpha reviewers notes) that I need a scene in between the one I’ve just finished, and the one I’m about to start.  If that’s the case, I write the new scene in my notebook, by hand.  Once the scene is typed in (after I’ve revised the entire novel), I’ll print out the scene and follow the above steps on it.

I look at each step as a bug fix.  Find the bug, fix it.  Make the fixes logical and in order.  Just not to the point where the reader can see your process.

Once the novel is revised, I sit down and make the fixes on the computer.  Sometimes I’ll make minor changes as I type in stuff, sometimes I won’t.  Too much fiddling isn’t good either.

Once that’s all done, I’ll read the story (out loud) again, make any minor (or major) changes I see, and ship the draft out to my Beta readers.

Wash, rinse, repeat.  Until you are happy with the work you’ve done.

I’d love to hear your comments on this.

During the move, I also upgraded the RAM.  Shortly after powering the server on, I noticed that any access to my harddrives (RAID or not) was extremely slow.  The kernel showed high load, with 98% being in system.  Wow.  Before I had time to realize it was the RAM I added, I lost a drive on the RAID array.  I pulled the RAM, and things went back to normal.  I added a new drive and started rebuilding the array.  At 97.1% complete a second drive got a permanent read error.  Good bye array.

I managed to get the array back up degraded, but I couldn’t rebuild it.  Now I know I could probably have used dd to copy the disk to a new one, get another new drive, and rebuild that way.  Oh well, hindsight is 20/20.

I needed to get my data off the degraded array.  There was no critical data on it, so it had never been backed up.  How am I going to back up 2+ TB of data anyway.  It mostly contained over 1000 recordings from MythTV, some audio and video, and pictures.  Still, if I could save the data, I really wanted to.  So, I went out and purchased a Drobo.  Drobo’s don’t really like Linux that much, so I hooked it up to my Mac Mini and started copying data over via rsync.  Wow, was that slow!

It turns out the Drobo is a really. slow. device.  Although it can handle watching live TV via MythTV, it barely keeps up.  Anything else done on the Drobo kills the TV.  Granted, the Drobo is a Firewire 800 device, but it still slower than that would imply.  Knowing that I will be recording and watching 2 HD streams at the same time, I knew the Drobo wouldn’t cut it for me.  For example, a Drobo rebuild could take days, and it’s all internal to the Drobo, no firewire used.  But, during rebuild, live TV was impossible to watch.

I built a new 4 disk 2.8 TB RAID5 array under Linux, and started transferring live TV and recording to the RAID5 array.  Things got much better  I decided to do some tests.

The Drobo, connected via Firewire 800, writing a 16GB file took 628.89 seconds (10.48 minutes) ~25 MB/s
The Linux RAID 5, writing a 16GB file took 135.59 seconds (2.26 minutes) ~121 MB/s

25 MB/s is doable for two HD channels, except for the issue that nothing else could use the drives at the same time.

I then tested the connection via the network.  My MythTV Server is a simple Pentium M 1.5 GHz system, and it uses remote Samba shares for it’s storage, over GB ethernet.

The Drobo via OSX SMB share, same 16 GB file took 1451.37 seconds (24.19 minutes) ~ 11.3 MB/s
The Linux RAID5 via Linux SMB share, same 16 GB file took 459.08 seconds (7.65 minutes) ~ 35.7 MB/s

Ummmm.  Ouch!  The Mac Mini is a Core2Duo at 2.6 GHz, the Linux is a Core2Quad at 2.6 GHz.  Neither used any noticeable CPU during the file transfer.

Tweaking the OSX smb.conf file didn’t help any.  What did help was specified here.  That alone changed my speed to 16 MB/s.  Still not good enough, but better.

The Drobo is a cool device, and has it’s place.  But not in  a media streaming type environment.  I’ll use the Drobo to store my audio, pictures, and movies, but not my live TV recordings for MythTV.

We’ll see if we can remedy that.  Wordless Wednesday is still around, just relegated to fewer time slots.