[triscabor]

I've been racking my brain on a seemingly intractable problem: the systematic automation of typical jobs. Okay, that probably sounds a bit crazy, but here me out.

My first goal was to aggregate a ton of supporting data to kick things off. I've done that by writing a few tools and applications to scrape numerous job sites and clean the data, then store it in a database.

The problem is, I'm kind of stuck -- I have all this nice data, but no real tangible way of moving forward. My initial goal consisted of two very different options:

1. Finding commonalities between discrete job tasks, and then mapping these to specific modules or processes to automate them. Then composing these into a single "job" entity. From an architectural perspective, this would be ideal. From a software engineering perspective, this is vital.
2. Recruit others in a crowd-sourcing application, either marketed as-is, or sold as another useful or entertaining destination.

These are not mutually exclusive.

I should note, these pre-empt the actual creation of an expert system -- that part would probably be a lot easier, using something like Drools for Java, or PyKe in Python. But this all hinges on having lots of domain knowledge, and knowing how to also translate it into business rules, software, etc...

Thanks for your input!

[Korrelan]

Hi Triscabor

I believe this is your first post, so welcome, also... I'm only on my third coffee so I'm not firing on all cylinders yet...

the systematic automation of typical jobs

Firstly by 'job' I presume you mean employment? Are you wanting to automate the actual job (by machine) or generate a list of physical/ mental requirements to do a specific job. Also this can mean any job from a road sweeper to neuro scientist depending on which circles the person replying to your query moves in. 

At what level/ resolution are you storing your data ie: 1 Turn up,  2 Do job, 3 Go home, lol, any job can be broken down into 'sub' jobs... and so on...

to specific modules or processes to automate them

If your just wanting to enter a job title into a system and retrieve a list requirements a person would need to do said job, then a simple relational database should do the job.

If your trying to create a system that can actually perform the jobs/ tasks, well... that's a slightly harder problem lol. 

[ranch vermin]

yeh I didnt quite understand either.

You want to automate a job allocator computer???   

If so.

a) make a 1024 bit descriptor for what a job is.  which could include information about the person that went for it.

b) then put them into categories automatically, by designating some kind of boundaries and then dragging the jobs into each other due to similarity, as they pop up to the system.

c) have a method that stands for if the person got the job or not, or if he did any good at it,  or got fired.

d) take the person data, then graft the perfect person for the job, with a genetic algorythm.

Maybe not that exactly,  but machine learning does stuff like that, im not sure what you are after to cater your functionality.

[triscabor]

Hi Triscabor

I believe this is your first post, so welcome, also... I'm only on my third coffee so I'm not firing on all cylinders yet...

Thanks! I wasn't sure if I was supposed to drop this in the welcome forum.

Firstly by 'job' I presume you mean employment? Are you wanting to automate the actual job (by machine) or generate a list of physical/ mental requirements to do a specific job. Also this can mean any job from a road sweeper to neuro scientist depending on which circles the person replying to your query moves in. 

At what level/ resolution are you storing your data ie: 1 Turn up,  2 Do job, 3 Go home, lol, any job can be broken down into 'sub' jobs... and so on...

Yep, any typical employed person's job -- for simplicity let's exclude those that are physical labor or entertainment oriented, and focus more on knowledge workers, administrative jobs, etc...

In terms of granularity, It would be best if it was very very granular. Toy example:

Given an example task "write a letter", a final product like this might be:

1. Pick up writing instrument and blank piece of paper.
2. Lay paper out flat on a smooth, hard surface.
3. With writing instrument, draw in a left to right fashion (assuming english or latin language), individual characters that form each word, followed by appropriate punctuation.
4. Continue writing until there is no space on the current line, then move down to the next line and continue.
5. Repeat this until the writing is complete. If there is no more paper left, go to step 1.

The physical motions should have no bearing on this question -- it's just a toy example to give you an idea of granularity.

If your just wanting to enter a job title into a system and retrieve a list requirements a person would need to do said job, then a simple relational database should do the job.

Nope, I already have this data, the means to continually scrape it, and the scripts to add it for use in a relational or document database.

If your trying to create a system that can actually perform the jobs/ tasks, well... that's a slightly harder problem lol. 

Yep, this is what I'm after.

[triscabor]

yeh I didnt quite understand either.

You want to automate a job allocator computer???   

If so.

a) make a 1024 bit descriptor for what a job is.  which could include information about the person that went for it.

b) then put them into categories automatically, by designating some kind of boundaries and then dragging the jobs into each other due to similarity, as they pop up to the system.

c) have a method that stands for if the person got the job or not, or if he did any good at it,  or got fired.

d) take the person data, then graft the perfect person for the job, with a genetic algorythm.

Maybe not that exactly,  but machine learning does stuff like that, im not sure what you are after to cater your functionality.

I think my previous post might clear things up. Your above idea is pretty interesting, my only concern would be how to GET that data -- requires personal employment information that would simply be private, legally confidential, and something most people would not share (and if they did, it's hard to determine if it's trustworthy or robust enough.)

[Korrelan]

So you need a system to break tasks/ jobs down into their base constituent parts...so a physical robot/ machine can perform them instead of a human.

If this is your requirement then I'm afraid there is no solution generally available to the public, many institutions/ corporations are working on the problem and it comes under the banner of AGI.

[8pla.net]

Welcome triscabor

My first goal was to aggregate a ton of supporting data to kick things off. I've done that by writing a few tools and applications to scrape numerous job sites ...

Reminds me of what I just built today, with somewhat surprising results... 

A chat bot tool kit to scrape a major search engine to get its responses.

Object Oriented classes are best, I think, but here is some pseudo code :

Code

/* 1. Pick up writing instrument and blank piece of paper. */

      function get(pen,paper){}

/* 2. Lay paper out flat on a smooth, hard surface. */

      function put(paper){}

/* 3. With writing instrument, draw in a left to right fashion (assuming english or latin language), individual characters that form each word, followed by appropriate punctuation. */

/* 4. Continue writing until there is no space on the current line, then move down to the next line and continue. */


      function print(pen,text,paper){}


/* 5. Repeat this until the writing is complete. If there is no more paper left, go to step 1. */


      var pen,text,paper;

      for loop {

      function get(pen,paper){}

      function put(paper){}

      function print(pen,text,paper){}

      }  loop

I hope this helps you think about your knowledge base and inference engine design.


_________________________________________________________________________

[Zero]

Hi triscabor,

I have all this nice data

Can we see a sample of what you've got so far please?

[ranch vermin]

Taking one of our common day activities and putting it into an exact step by step is a little daunting a problem,  i recommend taking a step back, have a think and try to find some unified way to solve a fair degree of problems with one method, (what im doing.) because the job is massive, handling seemingly simple day to day tasks - and could get needlessly complex.

[Korrelan]

If this is your requirement then I'm afraid there is no solution generally available to the public, many institutions/ corporations are working on the problem and it comes under the banner of AGI.

Perhaps my last comment wan't that constructive lol, I've been trying to get out of the habbit of spurting loads of information when not required.

Although a total solution to your requirements doesn't exist, this forum is a good place to get/ build ideas towards finding a solution.  We are all working on various incarnations of AGI's.

As you can see, loads of helpful input already.

If your long term goal was not achievable then we would all be wasting our time and efforts, so ignore what I said and start breaking the problem down.

A semantic tree perhaps like Ranch says, break each task down into it's constituent parts, start at a high resolution then work to finer detail eventually arriving at servo/ machine level instructions. Even building a simple parsed language to hold the order/ instructions that can easily be converted at a later date into any format as your ideas develop. Start getting the nuts and bolts down.

Code

Function Pickup_Object (x)
  Look_for (x) - (see below)
  Right_Grasp (x)
  Lift (x)
End Function

Function Look_For (z)
  Load_object_pattern (z)
  Seek_Surface
  Scan_for_pattten (z)	
End Function

Edit: Oh! which is just what 8plan.net said... I must read all comments before I spout lol.

[ranch vermin]

the hierarchy of functions seems like a good idea.  Maybe if you go to some level of low level, without actually making it all the way to moving the fingers/arm etc, could be a good way to start.

[8pla.net]

I ported an expert system C language source code to compile on the Linux operating system using g++ in a Bash Terminal.  For all you Windows users out there, g++ is a Linux C++ compiler.

Would you like me to post my working expert system C language source code listing as a foundation for study?   If not, maybe eventually, so I don't break the flow of this discussion... I'll start a new separate topic on programming an Expert System in the C Language on Linux.

Reference:  http://www.drdobbs.com/rule-based-programming-in-c/199702471?pgno=4

[triscabor]

Hi triscabor,

I have all this nice data

Can we see a sample of what you've got so far please?

The project lives heres: https://github.com/christabor/jobScraper

I don't commit data files to source, but you can run it yourself if you'd like. I've uploaded example json output here though

Industry (category) and job titles:
https://gist.github.com/christabor/217cb3fdd6c4fc96fe35

Single job title
https://gist.github.com/christabor/553fc8cd320c1359099e

[triscabor]

If this is your requirement then I'm afraid there is no solution generally available to the public, many institutions/ corporations are working on the problem and it comes under the banner of AGI.

Perhaps my last comment wan't that constructive lol, I've been trying to get out of the habbit of spurting loads of information when not required.

Although a total solution to your requirements doesn't exist, this forum is a good place to get/ build ideas towards finding a solution.  We are all working on various incarnations of AGI's.

As you can see, loads of helpful input already.

If your long term goal was not achievable then we would all be wasting our time and efforts, so ignore what I said and start breaking the problem down.

A semantic tree perhaps like Ranch says, break each task down into it's constituent parts, start at a high resolution then work to finer detail eventually arriving at servo/ machine level instructions. Even building a simple parsed language to hold the order/ instructions that can easily be converted at a later date into any format as your ideas develop. Start getting the nuts and bolts down.

Code

Function Pickup_Object (x)
  Look_for (x) - (see below)
  Right_Grasp (x)
  Lift (x)
End Function

Function Look_For (z)
  Load_object_pattern (z)
  Seek_Surface
  Scan_for_pattten (z)	
End Function

Edit: Oh! which is just what 8plan.net said... I must read all comments before I spout lol.

I've had the basic notion of decomposition in my head, but just the act of practicing and spelling it out seems to help. One thing that just recently occurred to me is the possibility of using Natural Language Processing to "assemble" the decomposed version (I believe the notion of realizing a concept into concrete code and data models is referred to as reification). In fact, I actually did a very naive toy version of exactly this for a separate project of mine: https://github.com/christabor/MoAL/blob/master/MOAL/software_engineering/reification/reification.py

The idea is to generate a parse tree and then using the different part-of-speech tokens, output actual code from each one. So, in this case, verbs map to functions and nouns map to classes. Unfortunately it's not very robust so the wrong nouns often map to a class, because it does not determine the subject correctly -- but this could be overcome with more robust code.

With this in mind, I could imagine taking all the "tasks" for which I have raw, structured data, analyzing the parts of speech for each discrete task, and then generating pseudocode for all tasks for all jobs. This could be a first stepping stone. The nice thing about this is that duplicate tasks across seemingly disparate jobs could become unified and make it easier to "divide and conquer" the problem space.

I think just talking about the problem is having a positive effect on my thinking!

[ranch vermin]

Your english wont be complete enough without ALOT of guesswork on behalf of the NLP algorythm, because when we explain things 99% of the activity is omitted from the sentence entirely.

This is an AGI attempt, and its an unsolved problem to date.

Again ill suggest what I suggested before - go to some level of specificy, but dont go any further at first,  itll be alot easier for anyone.