The histogram post

I mentioned that I was working on some histograms – basically looking for anything that might be abnormal so that it can be a small section in an otherwise large paper. There are a group of people who are doing some undeniably important and impressive work at my current institution. A lovely way to save what has otherwise been a very slow year would be to work my way in with some of their projects.

But I got this. A mess. Jumbled, bi-modal garbage. No idea what’s happening within those 4 groups, so I kept trying new methods. New normalization schemes. But I just kept getting crap. Which is frustrating when I’m working on being depressed and keeping up with TNT programming!

I had to go to the office yesterday – had a brief presentation to make at a smaller meeting, and decided to get some ideas on my histogram mess while I was speaking. So I added some slides, and went in and watched people puzzle over them.

It’s not going to work, I found myself thinking, and wondered briefly if that was coming from my being depressed or if I was just being realistic. One of the most difficult parts of being mentally off – for me – is not knowing if I’m seeing something badly or if my perception is fine and the situation is bad. Plus, I get so tired trying to figure it out that I just give up. Shuffle back home as quickly as possible so I can watch television and mindlessly run Matlab code on my histogram data.

A couple people had decent ideas though. Normalization techniques that made a bit more sense than my initial effort. Someone suggested taking a closer look at some results I may have discarded a bit too quickly. Yet another person suggested an approach I had specifically decided against using and I mustered the energy to explain why I thought that was a bad way to go. A bit more energized – and knowing I was running out of time to be involved with this manuscript, I returned home yesterday afternoon, snuggled back into pajamas and nestled on my love seat, and worked through more numbers and graphs.

After it got dark last night - probably right around this time - I loaded some numbers into Excel, closed my eyes and sighed. This was right – I had no question that the method was perfected and that if something was different between my 4 groups, I should see it. If I got a mess, there was some problem in the data that I couldn’t normalize out. Damn retrospective studies, I thought. It’s hard to go through data I didn’t collect and figure out what was different between year 1 and 4, if the groups were distributed appropriately through time, if some acquisition methodology had been better than others.

But I plotted the data, and found this.

It’s pretty. It looks as it should, and actually holds the exact result I hoped to see. I wanted that low, broad peak (in yellow) from the beginning and had – after multiple attempts and long weeks traveling, returning home, dreading this talk I’m to give tomorrow afternoon (insert shudder) – decided I wouldn’t find. I knew what I wanted that yellow peak to mean. I don’t know what that green blip is to the left of the peak, but I now have hopes of figuring it out. I wrote email this morning and explained what I knew. I received an enthusiastic response this evening from scientists with whom I hope to work closely in the future.

I’m relieved, obviously, and pleased. A bit surprised.

My point? Well, it’s the same underlying data. It’s all in how I look at it – where to focus my attention, how to normalize – if I need to normalize at all. Sometimes I can’t figure it out. I get to know the data, memorize which patients are in which groups so I don’t have to consult my lists so often as I go through long loops of guess and check to see what I might find. I had worked on this particular set of data for over a month. Just kept poking at it, creating new distributions, trying different bin sizes, looking at specific regions – some small, some large, some subjective, some automated – and came up all jumbled. I just couldn’t see it clearly.

I was a bit troubled that I went to the group meeting for help, though it was certainly appropriate. I do, after all, have a PhD and am working at a post-doc in a field very closely related to my graduate work. I should be able to figure this out, right? See the data as they should be? But I didn’t until I heard some ideas – sorted through some advice. And maybe that’s the point – I’m now at the stage of professional development where I know the right idea when I hear it. Can discard methods I think are poor, can quickly implement small changes into my existing code to fix problems I hadn’t considered in quite the right way.

The bright side is that I got through the data in about 5 hours after the meeting yesterday. All that knowledge – gained in a most frustrating way through trial and error, but obtained nonetheless – played in and allowed me to interpret data quickly, choose maximum values from the right spots, divide by the right values at the right times, average and understand where the end results came from so I could back up and do statistics after oohing over my pretty histograms in Excel.

I guess my hope is that I get to that point in life. That I can tell when I’m stuck and know which people to ask for help. Remember to be grateful that I've always had people with good intentions who are ready and able to offer their comments and questions. That I can take those thoughts and recognize which are effective for me. To note results of high quality when they occur. And to hope that the underlying data – the world in general, the people I meet, the decisions I make – really do make sense if I can just look at them in the right way.

In the meantime, the goal is to give myself a break sometimes. Perhaps when I can’t make sense of the data at large, I just need more time to consider it. Meet the people who might understand it. I can write on a blog and find great pleasure in comments and site stats and reading what some of you think on your own blogs. Perhaps, after all, this is part of figuring out how life makes sense for me – taking that jumbled mess I had in the beginning and hoping it ends up in graceful curves I’ve had pictured in my mind.