KBO encounters Options

[maschnitz]

(Software engineer here) That, there, is a failure of the Amazon AWS service Ice Hunters is using as their server backend. Or a misconfiguration to same.

AWS is basically a way to rent time on central computers. Unfortunately, sometimes, they go down. You also have to get the names exactly right.

[Holder of the Tw...]

Okay, thanks maschnitz.

Time to find something else to do tonight.

[centsworth_II]

Just got through about 150 more mostly problem free images and then hit another patch of unreadable ones. Gave up again after about twenty. I hope my marking those as "Simply terrible" helps weed them out. If I knew for a fact that it did, I might hang in there longer and mark more "terrible" ones.

I just started paying attention to the counters at the lower left of the screen. The count changed by 5 to 8 after each time I clicked "done" so I guess that was about how many of us were counting at the time. The count change then dropped to 2 or 3 just before I quit. I guess others were dropping out as well.

When you first start counting, the count really jumps around. It takes a string of 50 or so image views before the count settles down to a fairly constant change.
Edit: Or not. Sometimes the count flips back and forth between trend lines a few hundred counts apart.

[centsworth_II]

A near miss!


Close to you...

By the way, it's been clear sailing with nary a bad image on my current ice hunting expedition.

Two asteroids!


Overlapping KBO/VARs?


This is getting ridiculous! All this in one sitting?!

[Greg Hullender]

I find I'm marking about 1.5 objects per frame. From my own background in ML and data labeling (plus the note on the page to overlabel) I'm guessing this is just fine. It hugely reduces the space requiring followup, and that's probably all that's wanted/needed.

I have a bit more trouble with the labeling of image quality. I realize those are of only secondary value, but I do like to do things right.

--Greg

[nprev]

I've only been rating image quality if I get a streak of real garbage pics on the theory that it might be a bad batch. The occasional lemon...well, that just happens.

[nprev]

Here's today's best shot for me: asteroid & big honkin' KBO/variable star. John, question: Is there an expected detection rate for variables? Would like to know as a vector check for our respective discovery rates.

Attached thumbnail(s)

 

[Greg Hullender]

I'll bet that when you first start, they give you a bunch of pix that had likely KBO/Variables in them. That both lets you come up to speed and lets them get an idea of how good you are at it. Somewhere around 150 images, my find rate for KBOs dropped drastically. (I stopped at 300--my hand hurts.)

When I click the "My Icehuntings" link, I see that nearly everything I've marked has also been marked by 8 to 16 people, so I'll guess I'm on the right track.

The rules I developed for myself are more or less as follows:

1) Don't click really little ones.
2) Don't click if it has even a single black pixel completely inside it.
3) Don't bother with asteroids unless they're very obvious.
4) Just skip bad pages--don't try to say why. (Did I mention my hand hurts?) :-)

I think they ought to give us keyboard shortcuts for all the buttons; minimizing mouse movement really helps reduce RSI.

--Greg

[nprev]

I agree with all your rules save #3, Greg; I think that the faint ones are by far the most likely new discoveries.

[ngunn]

I'll bet that when you first start, they give you a bunch of pix that had likely KBO/Variables in them. That both lets you come up to speed and lets them get an idea of how good you are at it. Somewhere around 150 images, my find rate for KBOs dropped drastically.

Like many, I find this a really exciting project. I'm interested in how it was set up, and your surmises suggest that we may not have been fully informed about what we would be 'fed' and what would be done with our responses. If true, that's unfortunate. Are there things we don't know about how our input is being used? This is a technique employed by psychologists but surely not necessary here. So, NH team (and I know you are our friends), is Greg right?

[centsworth_II]

Somewhere around 150 images, my find rate for KBOs dropped drastically....

As you can see in my last post I just went through a really productive stretch, and that was as I neared 4000 images viewed. I've been through dry stretches and productive stretches and stretches of unreadable trash, but I have no reason to think it's anything but chance.

John Spencer said, "...we probably will add artificial objects in the 2011 images when they are posted- it's important to add artificial objects to the data to test what fraction of objects of a given brightness we can actually find."

So there may be artificial objects in some of the images, but I doubt that the images are fed in any particular order to any particular person.

[Greg Hullender]

If true, that's unfortunate.

Why do you say that? It's a very sensible thing to do. When people are new to the task, it makes sense to give them plenty of well-understood images. I'd expect to have to discard the first few dozen from each person anyway. People are great at this sort of task, but a) they take time to master it and b] they benefit from a little help.

Once someone is up to speed, then it makes sense to turn them loose on the less-well-understood stuff. But almost all human data labeling protocols I know of have an enrollment period. There's nothing wrong with that.

--Greg

[Greg Hullender]

I hope my marking those as "Simply terrible" helps weed them out.

I think these are really bad subtractions. What I want is a simple "bad image" button to indicate that I gave up on it. I might have marked one really obvious one, even on a bad image, but even then it should be taken with a grain of salt.

I also want keystrokes. E.g. press the spacebar for "done marking" and (maybe) press "X" for "bad image." Anything to minimize use of the mouse.

Given nprev's comment that bright asteroids will already be known, I've quit marking them entirely. They take too much time, and they're not what we're there for.

--Greg

[ngunn]

There's nothing wrong with that.

Nothing at all, if it's signposted so that participants are fully briefed about how it's working and when they're entering a different phase.

[centsworth_II]

Tic Tac Toe