tag:blogger.com,1999:blog-36768584.post6811300511183751653..comments2024-03-03T18:49:34.382-05:00Comments on Omics! Omics!: At the Edge of The CloudKeith Robisonhttp://www.blogger.com/profile/04765318239070312590noreply@blogger.comBlogger13125tag:blogger.com,1999:blog-36768584.post-68610894873134686082016-03-31T12:15:58.168-04:002016-03-31T12:15:58.168-04:00If you are repeatedly doing assembly work, you wil...If you are repeatedly doing assembly work, you will see need to keep in-house high RAM server.<br />If that becomes a case, I can share another tip that you may find useful.<br /><br />For about $1.5K, you can get HP G8 servers from ebay.<br /><br />http://www.ebay.com/sch/i.html?_odkw=dl360+g8&_osacat=0&_from=R40&_trksid=p2045573.m570.l1313.TR0.TRC0.H0.TRS0&_nkw=dl360+g8&_sacat=0<br /><br />These servers take 384GB-768GB ram.<br /><br />http://www8.hp.com/h20195/v2/GetHTML.aspx?docname=c04123167<br /><br />So, add another $1.5K and you are at 384GB.<br /><br />http://www.amazon.com/Kingston-Technology-16-PC3-12800-KVR16R11D4/dp/B0088SSUTO/ref=pd_sim_sbs_147_1/175-2649101-6950441?ie=UTF8&dpID=41jo1zCXVkL&dpSrc=sims&preST=_AC_UL160_SR160%2C160_&refRID=00JAFBT80FB4NDM8ETTH<br /><br />homolog.ushttp://homolog.usnoreply@blogger.comtag:blogger.com,1999:blog-36768584.post-42748017181354648612016-03-31T11:15:02.101-04:002016-03-31T11:15:02.101-04:00The tool may be new, but the short read assembly s...The tool may be new, but the short read assembly steps being used by it are not new. So, you have to replace the memory-inefficient steps with efficient steps. <br /><br />Please email me, if you need more help.homolog.ushttp://homolog.usnoreply@blogger.comtag:blogger.com,1999:blog-36768584.post-53912734114460403522016-03-31T10:57:09.110-04:002016-03-31T10:57:09.110-04:00Surprisingly, the tool I wish to play with is very...Surprisingly, the tool I wish to play with is very new, not very old -- though perhaps it has inherited inefficient structures from predecessors. Or at least not tried to use the very efficient ones developed for programs such as Minia.Keith Robisonhttps://www.blogger.com/profile/04765318239070312590noreply@blogger.comtag:blogger.com,1999:blog-36768584.post-62412169189192243512016-03-31T10:55:05.717-04:002016-03-31T10:55:05.717-04:00"But I wanted to play around a particular too..."But I wanted to play around a particular tool (to remain anonymous), and discovered it is recommended to be used with 500Gb of RAM."<br /><br />Most programs using 512 Gb RAM were written in an era, when short-read assembly was not properly understood. I do not understand why one would need 512 Gb RAM in 2016.homolog.ushttp://homolog.usnoreply@blogger.comtag:blogger.com,1999:blog-36768584.post-49614326729089397562016-03-31T03:51:14.961-04:002016-03-31T03:51:14.961-04:00Hi Keith
We have established the CLIMB project in...Hi Keith<br /><br />We have established the CLIMB project in the UK to help solve this problem for large metagenome assemblies for microbial genomics researchers. We offer up to 3Tb RAM machines at http://www.climb.ac.uk.<br /><br />Specific to UK only at the moment but we may be able to discuss collaborations if they are related to medical microbial genomics (new antibiotic discovery would count .. for example)<br /><br />Best<br />Nick<br />Nick Lomanhttps://www.blogger.com/profile/12121179953421841062noreply@blogger.comtag:blogger.com,1999:blog-36768584.post-5497469082302530222016-03-30T11:54:40.718-04:002016-03-30T11:54:40.718-04:00There's a blog post describing the G series he...There's a blog post describing the G series here<br />https://azure.microsoft.com/en-us/blog/largest-vm-in-the-cloud/<br /><br />If you go to the pricing page and select West US the pricing will show up:<br />https://azure.microsoft.com/en-us/pricing/details/virtual-machines/<br /><br />If you have a different region, it just says the G-series is not available in this region, which is unfortunately rather unhelpful.<br /><br />RaviAnonymoushttps://www.blogger.com/profile/11575312508256949389noreply@blogger.comtag:blogger.com,1999:blog-36768584.post-13529643816866557592016-03-30T11:31:50.368-04:002016-03-30T11:31:50.368-04:00Angel: Thanks -- X1 will be sweeeet when it comes...Angel: Thanks -- X1 will be sweeeet when it comes on-line - up to 2Tb of memory and 100 virtual CPUs!<br /><br />It did take a little poking to find those G instances for the reason you gave -- I would prefer it show the unavailable types greyed out (or better yet, a grid which showed which types are in which regions). I must say that of these vendors, I find Amazon's consolidated instance type grid the best.Keith Robisonhttps://www.blogger.com/profile/04765318239070312590noreply@blogger.comtag:blogger.com,1999:blog-36768584.post-62512818544020892592016-03-30T11:09:23.245-04:002016-03-30T11:09:23.245-04:00A part of the problem is that the development of n...A part of the problem is that the development of new algorithmic ideas into production-quality tools rarely takes less than 10 years. There are often many conceptual and theoretical problems to solve, before engineering and implementation issues even become the real bottleneck. If you want to solve a problem that was formulated only 5 years ago (e.g. due to new developments in sequencing technology), nobody in the world probably knows how to do it efficiently. Some people may have ideas, but investigating them will take years. Meanwhile, you're stuck in trying to solve the state-of-the-art problem with state-of-the-art hacks, which often require state-of-the-art hardware.<br /><br />Communication issues are another part of the problem. Bioinformatics jargon is so different from algorithms jargon that it often becomes a major obstacle to the spread of ideas between the fields. As an algorithms researcher working with bioinformatics tool developers, I often encounter that in my work. Sometimes I follow a discussion for a long time without really understanding it. Then someone states the problem in a different way, and the problem and its existing solutions are immediately obvious.Jounihttps://www.blogger.com/profile/08868620962601644633noreply@blogger.comtag:blogger.com,1999:blog-36768584.post-72942467681985185732016-03-30T10:24:06.464-04:002016-03-30T10:24:06.464-04:00Ravi, you beat me to it!
To be fair to Keith tho...Ravi, you beat me to it! <br /><br />To be fair to Keith though, the G series only shows up in the pricing page for specific regions they are deployed (https://azure.microsoft.com/en-us/regions/#services). I did not find documentation of all the possible instances. If there is such a link, that would be nice to post here.<br /><br />AWS did announce the X1 series coming in "the first half of 2016" but these are not yet available. https://aws.amazon.com/blogs/aws/ec2-instance-update-x1-sap-hana-t2-nano-websites/<br /><br />-angelAngelhttps://www.blogger.com/profile/09938400316307441118noreply@blogger.comtag:blogger.com,1999:blog-36768584.post-45783472054339116862016-03-30T07:49:52.332-04:002016-03-30T07:49:52.332-04:00My University-based sequencing core recently purch...My University-based sequencing core recently purchased six 512 GB 20-cpu nodes to complement our 256 GB nodes. Our cost wasn't that bad - about $10K each - but we do have to share them with other people. I suspect that the utilization of the 512G nodes will be similar to the 256G nodes in being not very efficient and full of jobs that do not require that much memory. But when you need the memory it is nice to have. Some programs just do not scale very well yet since they are touted as the 'best' tool for the job at hand then one wants to give them a trial thus it is handy to have those larger memory machines.<br /><br />I think part of the problem is the 'deeply furrowed' (as you put it in your last post) parts of bioinformatics where everyone thinks that they can write a better assembler/scaffolder/mapper/etc and while they may have a brilliant idea their implementation is lacking. Excessive or inefficient memory use, not being able to run on multiple machines and so on. They run their program on a smallish organism or with fewer reads, find out that the algorithm is "perfect", publish said program and then never polish it enough to run on bigger data sets.<br /><br />Oh well, the above is an early morning before coffee rant. Now to sit down and figure out why my most recent BESST run died on one of the new 512G machines. Hope this doesn't mean I need a 1TB machine!<br /><br /><br /><br />Rickhttps://www.blogger.com/profile/09945270829932237628noreply@blogger.comtag:blogger.com,1999:blog-36768584.post-48218088044620810872016-03-29T22:52:16.880-04:002016-03-29T22:52:16.880-04:00Ravi,
Thanks for catching that - Indid look at A...Ravi, <br /><br />Thanks for catching that - Indid look at Azure when researching this, but forgot about it while writing. Hadn't found that big machine though - now to look into itKeith Robisonhttps://www.blogger.com/profile/04765318239070312590noreply@blogger.comtag:blogger.com,1999:blog-36768584.post-83900003607380942292016-03-29T22:34:36.666-04:002016-03-29T22:34:36.666-04:00Yes, I have had similar wishes while using Google ...Yes, I have had similar wishes while using Google Compute Engine. But maybe at some point the tools (e.g. read mappers) and pipelines of tools will need to be designed from the ground up to use a good multi-machine framework. i.e. the capacity to generate more reads is outstripping the point where it makes sense to try to use single machines with more and more memory. But as usual in bioinformatics, I guess it all depends on the particular problem. I have been thinking about trying some bioinformatics pipelines on CoreOS/fleet/rkt...JDhttps://www.blogger.com/profile/07019071956581720990noreply@blogger.comtag:blogger.com,1999:blog-36768584.post-73291463640046616082016-03-29T22:19:28.575-04:002016-03-29T22:19:28.575-04:00You missed Azure, the G series goes up to 448GiB R...You missed Azure, the G series goes up to 448GiB RAM, 32 cores, 6 TB SSDAnonymoushttps://www.blogger.com/profile/11575312508256949389noreply@blogger.com