Hi all,

I mentioned to a buddy (who is also an editor) that we used subversion in our editorial process. He didn’t know what that was, and said that they used either this big nasty home grown system, or email attachments, to coordinate the editorial process.  He was incredibly curious about how we used subversion and what else we were using.

I started writing this kind of long email and then figured that others might be curious as well about the various technologies we use (or are moving to, etc) at Python Magazine, so here’s a quick list of tools we’re currently using:

• Subversion - of course, I’ve mentioned this. We view every email attachment as a problem to be solved. Email is a communication tool. It is not a file transfer protocol (no, really - it isn’t), and it is certainly not a collaboration tool. We have a very simple directory hierarchy on the server representing the various stages in the editorial process, from the initial, original submission as received by the author, all the way to the final PDF rendering of the entire magazine, and all parts in between. The final review of the magazine even happens in SVN. We have a ‘corrections.txt’ file that we all add to as we review the PDF, and when that file is empty, the PDF is moved to the directory representing “go to press!”
• Plain text - sometimes less is more. I’ve edited and authored using Word, OpenOffice, LaTeX, and a few other tools. In the end, plain text with extremely simple and minimalist formatting tags win the day by a long shot. Authors aren’t forced to use any particular tool or platform to write their articles, editors don’t have to wonder which version authors have, which language setting they were using, etc. We don’t have to wonder if our version control system will handle a binary format properly, and the files are smaller. It’s also easier to run scripts against them to do things like strip formatting, or selectively apply it given a regex or something.
• Google Calendar - We are notified the night before any article deadline, and the calendar is shared among the editors. Theoretically, the same calendar could be used to indicate that an editor is going to be unavailable or a tech reviewer is going to receive an article, but so far, it mainly reminds us of upcoming deadlines.
• IRC/Google Talk - We actually don’t send very much email to each other. Sometimes we talk on IRC about emails we received or need to be added to the cc list of, etc. Almost everything we do involves either IRC or Google Talk. Of the 50 or so people on the authors mailing list for Python Magazine, at least 40 have gmail.com email addresses, and so do all of the editors here, so even some of the author/editor communication is email-free. In addition, the Python Magazine IRC channel is irc.freenode.net/#pymag, and you can talk to editors and authors there. The only email that gets sent is:
  • subversion server updates,
  • users who need to mail info at pythonmagazine to ask subscription questions,
  • authors sending to editors at pythonmagazine to submit article ideas (we don’t take them on irc),
  • replies to threads, usually initiated by one of the above actions.
• PHP - yes, believe it or not, the main site is written in PHP. The publishing company (MTA) was originally formed around php|architect Magazine, which is a magazine about PHP. That was in 2002. Today, there are two language-based magazines. Some day there may be five language-based magazines. Certainly, we’re not going to maintain websites using 5 different languages! O’Reilly doesn’t do it, and they publish entire *books* on different languages (and platforms! and databases!) I was impressed by the Python community’s understanding in this matter. Lesser communities would’ve sent lots of hate mail.
• Python - Doug Hellmann (our tech editor) and myself (to a lesser degree, because Doug is far better at it) write any little tools and scripts we need using Python. Sometimes I think about writing Python scripts just to make Doug laugh. Don’t forget, I launched this magazine not because I professed any deep knowledge of Python. On the contrary - it was because I figured there were neophytes like myself who would like to know more, and advanced coders who would like to look into areas of Python outside their immediate area of expertise within the language.
• Adobe InDesign - InDesign is the main layout tool. Layout is like some spooky ethereal realm to me. I imagine other tools are used during the layout/design process, but I don’t honestly know what they are. I’m sure the layout team prefer it that way. It’s probably better if I just say “I’d rather see the title moved up and to the right” than to start trying to tell them how to use their tools.

Those are the tools I can think of off the top of my head aside from back end things like a relatively standard LAMP stack that runs the web sites, and which I also don’t have much of a role in maintaining. Of course, there’s also one big element of all of this technology that blows them all away: the people. Every single person is technical in some way. Me, the layout folks, all of the editors for the whole company that I’m aware of, and even our fearless leader are all technical people. Technology is a common thread that runs through the entire organization, and ties all of us together. It makes an enormous difference, and I’m proud to be a part of the team.

For the past ~6 weeks, I’ve been talking to people, getting buy-in, educating users and administrators, and generating copious amounts of project proposal and six sigma documentation presentig VMware Infrastructure as an infrastructure building and management tool.

There’s a whole manifesto behind this, but I’ll try to boil it down. Basically, this client has three sites, and the infrastructure needs to be consistent at all three sites. Also, ideally it would be overseen and generally managed from one site (there are obvious limits to this, but you get the idea). My thinking is that I have three choices:

1. Order/rack/setup/test hardware and software, stage system, install stuff, ship to site, where someone else racks machine and turns it on.
2. Assume and require that there is a senior enough admin at each site already to take care of all of that.
3. Decouple the OS image from the hardware altogether and just build an infrastructure server “factory” at the main site, and ship (read, scp or similar) to the VMware servers at the other sites.

I chose option three - but this is oversimplification and doesn’t go into all of the benefits.

So, I just found out today that my bill has made it through Congress, and my project now has legs (read: funding)!! When the project is complete (the first phase is to migrate the main site using this methodology, and replication to other sites is a later phase of the project), I’ll try to give a talk on it or something.

In the meantime, if anyone has thoughts on virtualized infrastructure, or if you’re doing something cool with this technology, please post your comments. I value your insight!

First, and most important, Python Magazine’s premiere issue has been unleashed!

I love this business. Doing all of the negotiations, the communications with authors, coordinating with layout and contract people, web administrators, tech editors, and the like, can get pretty chaotic. It’s sort of like managing…. a tornado. And, like a tornado, one second you’re not sure you’re going to make it, and the next everything is just fine. Finally, at some time after midnight yesterday, I looked up at my TODO list and realized that Python Magazine’s premiere issue was complete.

I didn’t know what to do. “Do I just… go to bed or something?” When you do what we at MTA like to call “marathon editing”, the moment you stop you get this weird sensation - like the one you get if you go roller-blading for two hours and then take a walk immediately afterward. Your body has to get used to *not* roller-blading.

About the First Issue

So, the biggest news associated with this first issue is that it is completely, 100% free. That’s right! You can go download the PDF at will, sans payment of any kind. This is big news for a couple of reasons.

First, the magazine costs money to produce. We pay our authors very competitively, and there are also editors, layout/design folk, and other people involved in the production. The business plan for producing a magazine where people get paid for what they do involves selling that magazine to (at least!) cover the costs. The idea that we’re giving the first issue away for free is, if nothing else, a testament to our commitment to and confidence in the product, as well as our stability.

Second, giving away the first issue means anyone can get their hands on it, read it, share it, print it, leave it in the coffee area at work, and do pretty much whatever they want with it. This means more people will see it, and see what we’re going after, and give us feedback so we can make it better.

Third, of course, we hope it means more subscribers, so that we can have the support we need to make the magazine better, and to give more back to the community by providing more services and whatever else we can.

Participate!!

Please send feedback! I want feedback! We all want feedback! Send it to editors at  pythonmagazine dot com

Also, write for us! Never wrote before? Don’t know where to start? No problem - drop us a line (editors at pythonmagazine.com) and let us know what your thoughts are, what you’re doing with Python, how it’s helped you, etc. We’re happy to help you develop the article idea, and get it ready for publication. If you’re intimidated by writing - don’t be. There are lots of authors who have already submitted articles to us who are doing really cool things with Python, have never written for a publication before, and their articles are being published - and they’re great articles!

Anyway, go get the first issue. Let me know your thoughts.

My wife and I don’t really celebrate Valentine’s Day. I’m not sure why. I think it started because, for the first couple of years we dated, we were a few hundred miles apart, so we couldn’t. Now it’s just tradition. Anyway, to make up for it, we go all out on our anniversaries. We had several anniversaries before we were married, but this is our second wedding anniversary, and my wife got me at 160GB black iPod Classic

This is the exact model I wanted. I did not want the iPod Touch. I think the iPod Touch is an insult to my intelligence to tell you the truth. I don’t know why Apple thinks I’m stupid enough to not realize that the iPod Touch is going to magically grow a phone over the next year or two. I don’t know why they think I should spend more money to get 1/10 the storage capacity of the iPod Classic.

The real kicker is that the iPod Touch, which has 1/10 the storage capacity of the iPod Classic, has a direct line to the iTunes Music Store, but the iPod Classic - the social icon around which people build their identities these days, and which has 160GB capacity, doesn’t. Seems like Apple could just put a wi-fi radio in the iPod classic, give it a direct connection to the music store as well, and watch the kids stand around on the playground listening to each other’s tunes and downloading a copy of their own for $.99. Hell, Apple could probably eat 75% of the cost of the data connection, further subsidize it through deals with the ISP, perhaps a small fee built into the initial cost of the iPod, or a $.05 surcharge on songs downloaded directly to the iPod or something, charge the customer *once*, for the iPod instead of a monthly data transmission fee or whatever, and still make money.

For $50 more than the 16GB Touch, you can get the iPhone. It only has 8GB capacity, but replaces two gadgets you may carry around all the time. Your phone, and your point-n-shoot digital camera. I’m not a fan of Apple’s stance toward users who would like to run software not made by Apple on the hardware they allegedly own, but if we’re just talking value and comparing the devices in their line, I have to believe the iPhone is a slightly better value than the iPod Touch.

Anyway, this was really meant to be a glowing review of my iPod Classic. It works wonderfully. I’ve already loaded a few months worth of home brewing podcasts, all the music that I care about (for now - more coming), a bunch of photos of my wife and daughter and stuff, and I’m all set to go. My wife must truly love me. At some point I hope to figure out why.

Today I spent the day completely breaking down and replacing the entire above-ground portion of my in-ground pool’s plumbing. I had some problems crop up with my multi-port valve, and I had to take a few things apart to troubleshoot and remove the pump as the source of a pressure problem (namely, there wasn’t any).

Without going into too much gory detail about the troubleshooting, suffice to say that, over the course of the past few weeks, I’ve spent a great deal of time in the pump house. More than I ever had to before. While I was in there I had some time to closely inspect the plumbing. While taking a break and sipping a home brewed scotch ale, I just sorta stared at each component in the line and thought about the logic employed in the overall project. By the end of my 5-minute break, I was not happy.

This plumbing was put in place by a contractor that was hired by the previous owner of this house. Thinking about that got me really steamed for two reasons;

1. I know the previous owner of this house, and he’s a great guy.

2. The contractor completely fleeced the previous owner :-/

The pipe coming out of the ground is 1 1/4″. It went downhill from there. Connected to the black pipe coming from the ground a 1 1/4 barb/thread adapter, and held in (in part) by a worm clamp. Actually - two worm clamps. And guess what screwed into there? A 1 1/4″ thread/barb adapter! Shoved onto *that* with two worm clamps was another length of the black pipe, which went to a very sharp, and *iron* threaded elbow joint which had a 1 1/4 thread/barb bit attached to connect the pipe. Yes, more worm clamps. From there there was an enormous, old, and broken brass or iron valve, and then we see our first bit of actual PVC - a union going into the pump.

Everything in that last paragraph has been completely replaced with PVC.

From the pump to the multi-port valve, there’s a PVC elbow that’s still in tact, but it had a simple barbed adapter, to which was attached 1 1/2″ vinyl tubing…. with worm clamps. The vinyl tubing connected in the same exact manner on the multi-port valve side.

Everything above was replaced with PVC, and the connections to the valve and pump are now either threaded or glued - but not barb fittings, and there are no worm clamps.

The return line (from the multi-port valve back to the pool), was, if you can imagine, even worse. A barb stuck out of the multi-port valve, and attached to that was the 1 1/2″ vinyl tubing - using worm clamps. The tubing attached to another barbed fitting on the other side that was screwed into another enormous brass or iron valve that was so broken it was not ever used. The valve actually sat on a milk crate, and I never really had any reason (or time) to consider why this is: it’s because instead of the valve being attached to some kind of rigid pipe with an elbow, it just sat on a milk crate and was connected to the ground pipe using….. vacuum hose.

Yes, that vacuum hose. If you had a pool growing up, you might’ve swung lengths of this stuff around over your head to hear the noise it would make. Or you might’ve tied up your little brother with it. The point is, it’s a toy. It’s not meant for this kind of application. It’s the wrong size, but of course, worm clamps to the rescue!

I know this is all kinda hard to believe, so I took a few pics of the “before” version of the plumbing during the takedown/troubleshooting phase of my project.

So above, Note that big valve - that’s coming from the pool into the pump. See all that black electrical tape? Yeah - that’s bad to see in a plumbing design. There was also some shrink wrap stuff on parts of the pipe that are under this table. You can also see here the infamous milk crate :-/

Here’s the other half - vinyl tubing, barbs and worm clamps. Oh my! In its defense, it sorta worked, and had the nice side effect of being able to see the water moving through the tubing. I guess that’s cool. Of course, my pump has a clear lid so you can see what’s going on, and there’s that nifty pressure gauge on the multi-port valve so….

This is a shot of my ‘testing’ setup. It’s a closed loop - the water comes from the bucket, up the hose on the left to the pump, and out of the pump back into the bucket. This worked wonderfully and allowed me to prove to myself that I didn’t need a new $x000 pump

The funnier part here is that you can now see clearly in the background that the return valve is connected (with a worm clamp!) to the black and white vacuum hose!

I told you all of that so I could tell you this

Nobody ever questioned this monstrosity of a setup. Somebody came in and set this up, and somebody else paid for it, and was happy to not have to think about it. It was clearly shoved together with whatever parts the guy had lying around. And “shoved” is really the perfect word here. The bottom line is that this was a fiasco. An adventure. It was not taken seriously at all. I’m starting to see things like this more and more in both my professional and non-professional work. People don’t take the work they do seriously. They don’t think much about their work. They’re here today, gone tomorrow, and they’ll get paid on Friday, and that’s all that matters.

This whole plumbing system is just that - a system. Systems require thought. A well-functioning system is almost always the result of some amount of thought and design and perhaps (gasp!) discussion. Just because they’re locked away in a place nobody ever goes doesn’t mean they’re not important. In fact, most things that are put out of the way are put there specifically because they *are* important.

If you build systems, please take the building of those systems seriously. If you’re inheriting or coming into an existing system, or managing the building of a system, question it. Question everything. Ask why. Ask what the alternative solutions are. Ask what is compromised by doing it one way as opposed to another. Ask was is commonly done in your scenario and why. Be curious and skeptical. Look for red flags. Should $.10 tin worm clamps be used in a plumbing system? Won’t they rust to the point where they are unusable? Certainly this can’t be the optimal solution? What are some alternatives? Odd - I’ve never seen vacuum hose used in that way, can’t we use something else?

I know this is goofy, but this is how my brain works: every single thing you can learn is likely to be analogous to either a) something you already know or b) something you can more readily wrap your brain around. The good thing about that is that it means you can pretty much learn every single thing

In this case, my pool plumbing system is analogous to a computing system infrastructure, inasmuch as both are systems, made up of many and varied components - and each component represents a decision that must be made, and a potential source of a future problem. Does the component fit? Is it made fo this application? Does it integrate well into the rest of the system? I could be asking those questions of almost any component of almost any system whether it moves bits or water.

Well, enough rambling for one night. There was a message in that somewhere, but it’s late, and I’ve been working with PVC all day.

1. To remove an offending known_hosts key without opening the file or using sed or whatever, you can just run ssh-keygen -R <hostname>, which will remove any and all keys for the given hostname (see man ssh-keygen for other options like this).

2. To copy files from a local directory with spaces in its name to a remote one with spaces also in *its* name, escape the space in the destination directory *twice*. I was moving VMware virtual machines from one server to another today and it bit me. Here’s the right way to do it:

scp -P 2222 *.vmdk “localhost:/var/lib/vmware/Virtual\\ Machines/ks/.”

Note that I’m also working in an environment where everything I do requires various ssh tunnels and other workarounds, which necessitates the -P argument, and explains why it looks like I’m ssh’ing to localhost

(Thanks to deryni for reminding me about ssh-keygen - I was looking for an ssh flag for this)

Sometimes I get questions from people about stuff that I’ve long since taken for granted. One is “what’s the difference between mtime and ctime?”

The answer is simple, but I wanted to post it here in case it can help anyone. In UNIX:

mtime is “modification time”, and it is the time at which the last modification *TO THE CONTENT* of the file was made.

ctime is “change time”, and it is the time at which the last modification *TO THE PROPERTIES* of the file was made. By “properties” I mean things like ownership and permissions.  You can think of this as “chtime” - the last time a “ch*” command like chown, chmod, or chgrp was performed on the file.

I started a new job about 6 weeks ago. I’m now doing infrastructure architecture at http://gfdl.noaa.gov

GFDL stands for Geophysical Fluid Dynamics Lab. It’s a NOAA site that supports atmospheric and climatology research. So in other words, the work I do supports research into things ranging from global warming to what the atmosphere on Mars is like to the weather here on Earth to simulations of the shape and movement of Katrina. I think of it as sort of an Institute for Advanced Study devoted to climatology research. Great minds in the field are here.

The research actually takes place at three different sites, DC, Boulder and Princeton, and affiliations with academic institutions flourish as well. In fact, I knew at least 4 people who worked here because of interactions between this site and cs.princeton.edu, my former employer.

My job, as it’s been described to me, is to provide a vision as to the design and direction of the infrastructure which supports the rather enormous high performance compute (HPC) cluster. This involves something of a learning curve to understand what’s here, how the systems are used, what the needs are, what people like and hate, where the redundancies and inefficiencies exist, etc. It also involves having meetings and coordinating with people who manage the network, the facilities (power & cooling, etc), the security policy, etc. I’ll be grilled on my ideas, and create prototypes and demos to get my ideas across. Lots of communication.

An aspect of my job will also involve getting my hands on the HPC clusters themselves as well, which are also at each site. All of the clusters are on top500.org last time I looked. Just go through the pages and search for GFDL and/or NOAA.

The systems here are all Linux. Even the standard-issue workstations are running Linux.  Scripting is done in Perl and shell, but Python is everywhere, so I’ll be doing either Perl or Python if I have the choice (because “shell” == “csh” here, which I never took well too, honestly). Some aspects of the environment are pretty fascinating. For example, how exactly do you store (*and* easily retrieve, on the fly) 9 PETABYTES of data? How do you back that up? How do you recover from hiccups? How do you instrument systems consisting of thousands of CPUs,  to pinpoint problems and get them fixed? And, by the way, how’s the best way to tune a system’s network stack to use a 50MBps pipe (that’s Mega *bytes*) efficiently enough to move multiple terabytes of data every day between collaborators at different sites? How, exactly, do you consolidate services and provide failover across geographically dispersed sites?

So that’s it for now :)  It’s too early to tell how things are going, really. It’s certainly not the cushy environment that Princeton U. was, but there are bigger challenges and problems to be solved here, and that’s the part I’m looking forward to.

I relinquished control of my house to my friend and neighbors for the coming week, and I’ll be in Martha’s Vineyard until August 27th with my wife and daughter, along with my mother- and father-in-law, sis-in-law, and her fiancee. We have lots of activities planned, including tuna fishing (if they’re running).

So, if you don’t see me on IRC, or I don’t reply to your email, or I don’t show up to a meeting, or whatever, that’s why

I just want to ask the folks who do the marketing for Gatorade to stop coming up with beverage names that border on being downright threatening.

No, really.

I went into the refrigerator to look for something to drink just now, and way in the back I noticed a bottle of something I’d never seen before. I was pretty sure I didn’t want it, because it was blueish purple, and I’m not aware of anything that color that I’d want to ingest, but I wanted to know what it was, anyway. I noticed it had a Gatorade logo on it, and so I figured maybe it wouldn’t be so bad, because nothing Gatorade makes is anything like a normal color, and some of it isn’t bad. But then I saw the name of this stuff:

“Fierce Grape”

Now, for my money, I say give me some peppy grape, or maybe something a little stronger like “heavy grape” or something. If I’m feeling spry I might go so far as “aggressive grape”. But even there, there are still a whole bunch of levels of grape you have to pass through before you get into the same ballpark as the grape that is called “Fierce Grape”. You leap straight past even “intimidating grape”, and even that is too rich for my blood. Certainly no human would consider introducing one’s innards to something more grapy than that, would they?

There’s just no real need for my grape to be fierce. What kind of person needs their grape to be fierce I wonder? Have people gotten so aggressively competitive in every aspect of their lives that now even their beverage of choice has to be somehow more kickass than everyone else’s?

Well, I’m now relaxing with a Poland Spring, and I’m staying away from the dangerous-looking drink in the back of the fridge with the chip on its shoulder.

So, it’s been 1 month since we officially announced Python Magazine’s existence. I’m very happy (and even more relieved) to report that things are going wonderfully so far.

In these early days prior to the actual release of the first issue, the focus is primarily on content development. This is also called “getting articles in the door so we can publish a magazine”. The goal here is to avoid the scenario where you wake up on September 1, say “hey, I need content for the issue coming out on October 1!”.

It’s nearly impossible to attract authors, get good proposals, get authors to write articles, edit those articles, validate the code samples that come with the articles, format the articles, and compile them into a magazine you’d actually want to read… all in 30 days. This is why magazines (and even online publishers) shoot for a backlog of articles. Depending on the size and scope of the publication (and some other stuff like how time sensitive the material is), that backlog can be anywhere from 1 month (for online stuff, mostly) to a year. We’ve chosen a magic number of 4 months.

The initial goal was to commission 4 months worth of content before September 1. This means we’d like to have signed contracts with willing authors agreeing to write at least one article for us. As it turns out, we not only commissioned 4 months worth of content by *August* 1, but we’ll probably have almost 4 months worth of content *delivered* by September 1.

And darn good articles they are! Keep an eye out on the Python Magazine web site for a couple of things of interest:

• An editorial calendar - this is where you can see what kind of content is coming up in future issues of the magazine.

• A sample article! Sometime before October 1, we’ll probably post a sample article that is more or less representative of the content we’ll have in the magazine. I have my eye on an article we might use for such a purpose now - but you’ll have to watch the PyMag site for updates.

As usual, if you’d like to write for Python Magazine, drop us a line, and/or join the authors mailing list by sending a blank email to authors-subscribe at lists dot pythonmagazine dot com

Ok, I have lots of blogging to do now. I haven’t blogged in at least a week, and plenty is happening.

First, I have a new job. For those who didn’t know, for the past 6 years I was working for the Computer Science Dept. at Princeton University. I’ve moved over to work in an environment that will provide me with challenges that simply cannot and will not exist in the CS environment. It’s the hardest decision I’ve ever made, because Princeton is such a wonderful place to work that most people never leave.

Anyway, I now work here –> http://www.gfdl.noaa.gov

So, *another* person asked if I’m “on twitter”, so I went and . Whatever. As far as I can see, it’s a useless distraction, but people seem to be obsessed with it. I’m convinced that these people don’t have any real work to do.

Once you sign up, the only thing you can really do is send email to your friends to recruit them. How egomaniacal do you have to be to send email to all of your friends saying “hey, come and see what I’m doing at any moment in the day - because I *know* you’re interested in that”…..?

Maybe there’s some hidden useful purpose that I haven’t found yet. I’ve at least set up the IM gateway so I never have to go to the web page. More on this as I dig.  In the meantime, if you’re doing something with twitter that’s actually useful, please let us all in on it!

UPDATE: Well, that didn’t take long. My solution works, but there’s a better way. Create the connection in main(), and create separate *cursors* for each Host. Cursors are cheap, and you reuse the connection. Thanks to Brend on #python (irc.freenode.net) for the enlightenment.

Python has been a wonderful language to get to know so far. However, one thing I didn’t really miss about Java and C++ were the decisions that are kind of forced upon you when you have various objects working together in a program, controlled by code in a “main” function. Here’s one decision I was faced with that took more thinking than I’d like to admit to make.

I have a class, we’ll call it “Host”. Of course, there are also methods for that class, like Host.record_name() and Host.update_mac() and the like. These methods are just wrappers around some SQL.

I also have, of course, a “main” function, which is where we create instances of Host and call the methods we need on the objects.

The question now is, where exactly should we create the handle to the database? There would appear to be 4 choices (without getting overly absurd):

1. In main() and then pass the handle to the class’s ‘init()’
2. In main(), but pass the handle to each method of any object that needs it.
3. In Host.init(), so methods can refer to it without creating it themselves.
4. Inside the individual methods themselves.

I chose option #3. The only part that bothers me about it is that a new “Host” is created, and then destroyed, for each of a couple thousand lines of a file. Hence, there is the overhead of setting up and tearing down a database connection a couple thousand times every time this script is run. I’m not sure (yet) what the cost of this will be.

Certainly, the cost will be less than if I did it for each *method* that was called, so that insures that option 4 is probably not the right way to go.

The semantics involved in creating the handle in main() and passing it to Host’s init seemed less than straightforward to me. It seemed there was the potential for main to pass in a copy of the handle at object instantiation time, have the object be destroyed when it’s done doing its work, taking the connection with it, and then main still thinks it’s a useful handle and tries to pass it to the next object that needs to be created. If that’s not the case, then what is the status of this thing that main created? What will happen if main tries to pass it to another object?

Of course, this problem would also exist if we passed it to each method that needed it. It would also be more overhead, and it would seem needlessly draconian, especially since *every* method of the object in question would need it.

If I’ve misunderstood something, or have made a poor choice, feel free to clarify/flame/enlighten me in the comments

Technorati Tags: python, database, programming, development, sql, internet, technology, opensource

Social Bookmarks:

I give a talk that advocates writing. I do this because I owe my career to people who wrote down what they knew and made it available in one form or another; either free on a web site, or in a book that I bought.

When I started editing, one of my jobs was to encourage people to write for whatever publication I was working for at the time. I’m now Editor in Chief for Python Magazine, and I’m back in the role of encouraging people to make their knowledge available for others (including me!)

Python Magazine is going well in terms of authorship, but with all of the Python coders out there, I should be overwhelmed with article proposals! When I talk to people who clearly know what they’re doing, or are doing something cool, and ask them to write, the responses are, by now, pretty predictable - and mostly based on misunderstandings, myths, and other untruths. I thought I’d take a few minutes to address the most common ones here:

“I can’t write”

The king of reasons for not writing is this one. The problem isn’t really that the person literally cannot write. The problem is twofold: first, the person has absolutely zero confidence in their ability to write. Second, the person doesn’t understand what an editor is supposed to be doing to earn his keep.

I’ll tackle the second part of the problem first: an “editor” is not some gray haired guy with a cigar screaming about deadlines and rejecting 99% of everything that hits his desk. An editor is someone who works *with* writers. It’s not *supposed* to be an adversarial relationship - and it *is* supposed to be a relationship. It’s supposed to be a working relationship where the writer tries to relay concepts in a way that the *editor* can understand. In return, the editor asks questions, pokes, prods, and makes suggestions that help morph the article into something the *readers* will understand.

Further, the relationship may begin before a draft is ever submitted. I’ve helped lots of writers develop their *abstracts*, so that by the time they sit down to write the article, they have some clear idea where they’re headed. And the word is *helped*, not *dictated*. It’s not me passing back a marked-up copy and saying “fix this and do that”. It’s me writing back saying “I’m a little confused in this part, how does this work?” or “can you provide a use case in which someone would find this useful?”

Finally, this excuse may have foundations in a perceived language barrier. If English is not your first language, you may feel like you’ll never get published in an English publication. Not true. If you have the knowledge, and can write enough English to relay the concepts, a good editor will work with you to develop it into something suitable for publication. It may take a bit more time, but that’s usually not a problem. In technical publishing, knowledge trumps prose every day of the week.

“I don’t know what to write about”

One day while you’re coding, just stop. Stop and take stock of what you’re working on. In all likelihood, there’s an article in there. If you’re thinking “there are a thousand articles about this already, in addition to the online docs”, you need to know three words: “Fresh, relevant content”.

A thousand articles have been written about just about everything! Why do you think that is? In the tech world, it’s because technical people tend to pay close attention to the date of publication to figure out if the content is fresh and relevant to the current version of whatever they’re using. Because technology evolves, there is often a need to get updated information to the readers.

The other part of this objection is the idea that the reader can get the same information from online documentation, or that what you might write about is just too easy for anyone to find useful. This is usually based on an assumption that because the writer learned it in an hour reading online docs, that everyone can/will/wants to learn that way. I hate online documentation, especially for programming languages. So do lots of other people. That’s why people buy enough technical magazines for my publisher to let me head up another one!

Also, people read about things in magazines that they may otherwise never read about if it were online, and they get inspiration from these things on occasion. Why? Because it’s there. When I go to get my oil changed, I bring a magazine with me to read while I’m waiting, and I read stuff I wouldn’t normally seek out online because I have time to kill. When you see people at the DMV (or MVS, or wherever people go to get their licenses renewed or cars inspected), a lot of them have magazines with them. They have time to kill. Inspiration can come from the strangest places. So whatever it is you’re doing, write about it, and inspire someone to take whatever it is you’re doing and do something else cool with it. Even if it’s, say, the Python tempfile module or something mundane like that

There are three things I tell people about where to find article ideas:

• Write about stuff you know/do - this is a gimme.
• Write about stuff you *want* to know/do - You don’t have to be an expert on Twisted to write an article about it. If you’ve done some network programming with Python at all, and are wanting to get into Twisted, research doing what you want to do with Twisted, figure out if it’s a good choice, do a proof of concept, take some notes, and write an article based on what you’ve done!
• Write about stuff that is not done, or often done wrong - Best practices documentation is severely lacking. There are a trillion articles that talk about different ways to do similar things, but very, very few articles that say “Here are three common ways this is done, and here’s why this fourth way might be better” or “Here are the pros and cons of doing this using 2 different methods”. Because best practices documentation is lacking, things are quite often done in a suboptimal way. Articles like “How not to do web programming with Python” might cover why you don’t want to use the split() function to parse URLs, for example

More on the way

I’ll expand on these and address more common reasons for not writing in future posts. In the meantime, if you’d like to give writing a shot, and you write a decent bit of Python code, I invite you to come and write for Python Magazine. I look forward to working with you!

Technorati Tags: python, publishing, writing, technology, internet, editing, productivity,

Social Bookmarks:

I was interviewed by Tony Mobily of Free Software Magazine! You can see the interview here.

Also, if you’re tired of Slashdot and Digg, Tony also works on Free Software Daily.

So I was on more than one IRC channel today asking a question that I got lots of answers to, all of which looked really messy to me, so I dug into the documentation and found that in Python 2.5, strings have a new method, called “partition”.

Here’s the trouble I had, and why “partition” helps:

>>> m = “parent:child”
>>> x = m.split(”:”)[1]
>>> x
‘child’

This is actually pretty close to what I want. I want to split m, and assign the second element in the resulting list to x. The problem arises when there is no “:”. In that case, that second line is actually trying to assign to x by referencing an index that is beyond the end of the list. You get an “index out of range” error.

So you’re left with a number of options. You can use an “if” to check for the existence of the colon.

if “:” in m:
x = m.split(”:”)[1]

I guess. But then I have to use another “if” to check and see if x was ever defined, or I have to initialize x ahead of time. At this point I’m onto at least three lines of code for something that takes 1 line in Ruby, Perl, or PHP. That just *can’t* be right!

You can also go ahead and just add the “:” explicitly to “m”:

>>> y = m + “:”
>>> y
‘parent:’
>>> x = m.split(”:”)[1]
>>> x
”

Nothing wrong with that except I still think I just shouldn’t have to do that. I should be able to, as a last resort, go with regex and do a one-liner that assigns everything after “:” to x, and if there isn’t a “:”, then x would wind up being empty. Something along the lines of x = re.search(”:.*”, m) - or something like that.

Well, partition solves the problem rather nicely. From the documentation:

Split the string at the first occurrence of sep, and return a 3-tuple containing the part before the separator, the separator itself, and the part after the separator. If the separator is not found, return a 3-tuple containing the string itself, followed by two empty strings. New in version 2.5.

Here’s what I did with it:

>>> y
‘parent:’
>>> x = y.partition(”:”)[2]
>>> x
”

Of course, I got an empty string in this case, but “partition” did what I wanted “split” to do - namely, assign ” or None to “x” if the separator didn’t exist. I guess what makes this possible with “partition” is the fact that it returns a fixed-length tuple no matter what, while split returns an arbitrary-length list, which would introduce ambiguities in cases where multiple splits are done to a string.

Technorati Tags: python, python25, scripting, sysadmin, technology, internet, opensource, programming, development

Social Bookmarks:

CMP recently announced that they will cut 200 jobs, and shut down more than one magazine in the process, folding their content into other existing magazines. There has been a lot of buzz in the sysadmin community (which I know largely as a loose collection of people who belong to LOPSA, SAGE, USENIX, or local groups of various kinds) about how sad and unexpected this is and how it’s a sign of the times or something.

You Can’t Sell a Generalist’s Mag to a Market of Specialists

I actually think SysAdmin’s demise was a long time coming - in part because, well, I’m involved in the publication of a couple of magazines, and read lots of others, and know people who work in publishing on other magazines as well as large online media outlets (some of whom I also write/edit/consult with). The plain fact of the matter is that it is extremely difficult to cover a topic like system administration in a generalist sort of way when your audience no longer consists mostly of generalists.

For example, the last two issues of SysAdmin magazine I remember getting were about Database Management, and Linux, respectively. The database management issue talked about Oracle and MySQL, and then had 3 or 4 articles on things not really directly related to databases at all, if memory serves. The Linux issue is probably of no interest to the admin who was all revved up for the Oracle articles, because that admin is probably not so much a sysadmin as an “Oracle admin”. Meanwhile, the average Linux administrator is probably uninterested in the Oracle RAC Primer.

Most admins aren’t generalists anymore unless they work in academia, research, or a company small enough that there are only a couple of people to handle the entire infrastructure. Even people who would like to be more general aren’t doing generalist things in production. I know several people who work *only* on {Oracle, sendmail, websphere, whatever-other-service-you-like} at work, and nothing else, but they run Apache, Bind, Postfix, and a few other services at home. That knowledge is nice to have, but it’s hardly something you can use to market yourself as a production generalist administrator.

So the market is flooded with mail administrators, backup administrators, storage administrators, cluster administrators, network administrators, database administrators, websphere administrators, exchange administrators, desktop administrators… the list goes on and on and on. The magazine market has mostly followed suit. There are magazines about cluster computing, DB2, AIX, SQL Server, Linux servers, Windows servers, and lots of other specialized areas. Note in that list that there are two separate magazines for two separate database products, but no “DBA’s Journal”. In killing SysAdmin Magazine, CMP is just following along with market trends.

Programming has mostly gone in the same direction. Dr. Dobb’s Journal is no longer available on your local bookshelf, but you can find php|architect, a magazine about nothing but PHP, doing well. Others exist for .Net, Cold Fusion, C/C++, Java, and now even Python. I’m sure Ruby and Lua aren’t far behind.

“I know, let’s start our own magazine!”

There are lots of naysayers out there who seem to think you need a huge staff, tons of money, and loads of other resources to start a magazine. Not true. You need a few dedicated, motivated people, and a small amount of seed money, and some time and hard labor.

An example is php|architect. It started with two guys: the publisher (who was a software development consultant as well), and his business partner, who also knew Quark pretty well - plenty well enough to do the layout. Then I came on board. So after the first issue came out, the rest were all tech and copy edited by the publisher and I, and the layout was done by the other guy. That’s really minimizing the amount of work we did, but the point is you don’t need an army.

When you’re as big as CMP and you’re looking at cutting a million bucks or something from your operating expenses, you go ahead and cut away! A small outfit isn’t playing with these kinds of numbers. A full-time editor at a company like CMP probably makes a 6-figure salary. A guy like me who edits as a side job makes something like… far less than that. At that rate, you can add another editor, a tech editor, a couple of columnists, all part time, and the advertising revenue will still cover the costs.

So if you want to see a sysadmin magazine, get some dedicated people who have half a brain and go do it!

Good luck.

Technorati Tags: publishing, sysadmin, magazines

Social Bookmarks:

I have a confession to make: For the past 6 weeks, I’ve been leading a secret double life. By day, I’m a mild mannered system/network/database admin in academia. I also write some PHP, Perl, and Python code. By night, however, I’m an author and editor. My latest project is bigger than most. In fact, it’s an entire magazine. Devoted to Python.

I am the Editor in Chief of the newly launched Python Magazine.

Why on Earth Are You Doing This?

Python Magazine was created as a result of some rather unfortunate events in my own early experiences with Python. Getting started, of course, couldn’t be easier. It was what happened after I had been coding for a while that I had issues with. Once you needed to do something a little out of the ordinary with the language, it was hard to feel confident that the way I was going was the right way.

For example, I decided to wrap up a bunch of SQL calls in Python and expose them as an API using Python’s built in SimpleXMLRPCServer. I thought this was great, because then I could maintain a single back end API, and any language that could make an xmlrpc call could use it without me having to maintain APIs in several languages. Nice in theory, but people smarter than I questioned my decision to use the built in SimpleXMLRPCServer. The right road to take, though, was completely unclear.

As another example, I needed to get up to speed on using the python-ldap module, but found that a lot of the documentation lacked anything but the most basic of features, but I was trying to write a full-fledged LDAP management API (and accompanying command line and GUI tools). Other articles I found were outdated enough that people warned me not to bother with them, pointing to glaring issues with the code samples (which turned out to be true - some of what was in the code samples turned out to be completely deprecated!).

When I wanted to write code against a PostgreSQL server, the correct module to use was also not immediately obvious, so I had to hunt down the sites of various modules, see which ones were maintained, search for articles that weren’t 5 years old on how to use them… Gah!

What I really wanted was a resource that fed me information in a way that my brain likes to feed on information. I really wanted to learn to do things with Python the way I learned to do things with Linux, Solaris, PHP, and even non-technical things like photography, billiards, and brewing beer. I wanted a magazine.

There was no magazine. I was bummed.

How did you finagle this one, jonesy?

I have a friend named Marco Tabini. He’s a publisher. He runs Marco Tabini Associates in Toronto. He is the publisher of php|architect Magazine. He’s also a total geek. For fun he does things like writing lexical parsers… in PHP. Nobody should ever do that. He thinks it’s fun. I say pass on that if you are given the chance.

Marco and I met via email. I wrote to tell him that I had received my first issue of php|architect, and would not recommend it to a friend. I had found something like 15 errors (typos and grammatical issues) in the first two pages of the magazine. Marco wrote back and said “hey, we’re a small outfit. We’re an Italian immigrant and an Iranian immigrant, living in Canada, trying to edit technical articles written by people from all over the world with varying levels of experience with English… all for a largely American audience. Come help us out!” So I did.

Shortly thereafter I became Editor in Chief of php|architect. Now there were three of us. Oh joyous day.

Those were great times, and the magazine has since spawned its own online and on-site training, its own line of books, its own series of conferences, and even a cruise! It probably has stuff I don’t even know about because I haven’t worked directly for that particular publication since 2004.

The success of that magazine gave me the courage to go to Marco about 6 weeks ago and ask about letting me head up another magazine, this time about a topic of *my* choosing. We chatted on IRC for several hours over the course of about a week, bought a couple of domain names, settled on budgets and team members and all that, and set out to make Python Magazine a reality.

So… How’s it going?

Things are *REALLY* rolling now. There are columnists, there are tech editors, there are authors. Articles have been commissioned. Logos and trademarks are in place. The design team is rocking, the contract team is rolling, and the emails are flying. In the background, the sound of constant typewriter activity can be heard, just like on those old newscasts from the old Cronkite days. Exciting times!

That said, we still need LOTS of content. The behind-the-scenes of a magazine is that you’d really like to have something like 4 months worth of content “in the can” before “Volume 1 Issue 1″ is released. I’m convinced that this has never happened in the history of publishing, but it’s a great goal to have, and I’d be pleased as punch to be the first person ever to achieve it

If you’re a writer who is doing or has done something interesting with Python, or can illustrate high level concepts from the fields of computer science, research computing, or IT, using Python, we’d love to hear your thoughts!

Import This

In the end, I hope I can be a good steward to the language and community. I’ve already been in touch with a lot of wonderful people - authors and others - who’ve helped out in some way, either with the magazine, with my own buggy Python code, or both. That’s all the news that’s fit to print for now, but keep an eye here and on the Python Magazine website for more updates as they happen.

Oh yeah - and if you subscribe now, you get a discount, and a chance to win a MacBook!

Technorati Tags: python, programming, development, internet, technology, magazine, publishing, pythonmagazine, scripting, sysadmin, opensource, freesoftware

Social Bookmarks: