{"id":3355,"date":"2018-06-01T21:18:34","date_gmt":"2018-06-01T21:18:34","guid":{"rendered":"https:\/\/max-drake.cc\/?p=3355"},"modified":"2018-06-01T21:28:06","modified_gmt":"2018-06-01T21:28:06","slug":"neo4j-upload-from-csv","status":"publish","type":"post","link":"https:\/\/max-drake.cc\/?p=3355","title":{"rendered":"Neo4j upload from CSV"},"content":{"rendered":"<p>I had an excel sheet with some property data. I isolated it to one property for this exercise.<\/p>\n<p><img decoding=\"async\" class=\"alignnone wp-image-3394 lazyload\" data-src=\"https:\/\/max-drake.cc\/wp-content\/uploads\/2018\/06\/imn12-1024x627.jpg\" alt=\"\" width=\"1585\" height=\"970\" data-srcset=\"https:\/\/max-drake.cc\/wp-content\/uploads\/2018\/06\/imn12-1024x627.jpg 1024w, https:\/\/max-drake.cc\/wp-content\/uploads\/2018\/06\/imn12-300x184.jpg 300w, https:\/\/max-drake.cc\/wp-content\/uploads\/2018\/06\/imn12-768x470.jpg 768w\" data-sizes=\"(max-width: 1585px) 100vw, 1585px\" src=\"data:image\/svg+xml;base64,PHN2ZyB3aWR0aD0iMSIgaGVpZ2h0PSIxIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciPjwvc3ZnPg==\" style=\"--smush-placeholder-width: 1585px; --smush-placeholder-aspect-ratio: 1585\/970;\" \/><\/p>\n<p>The property has A Site, A block, A Unit and 8 rooms and 3 cupboards (denoted by room name they were in with a letter &#8216;a&#8217; at the end)<\/p>\n<p>In the first part of the process I set up Nodes for all of these elements.<\/p>\n<p>In this exercise I wanted to take data from a CSV file to populate the Graph Database.<\/p>\n<ul>\n<li>Ground items such as letter boxes, fences etc, I wanted to link to the site node.<\/li>\n<li>Exterior works such as roof, exterior cladding and exterior painting etc I wanted to link to the Block node<\/li>\n<li>The Unit node seems to be a placeholder for the rooms as the unit does not seem to contain any data in the spreadsheet<\/li>\n<li>All of the elements in the rooms need to be linked to the specific room node.<\/li>\n<\/ul>\n<p>I used <a href=\"https:\/\/neo4j.com\/developer\/guide-import-csv\/\" target=\"_blank\" rel=\"noopener\"><strong>this article<\/strong><\/a> as a resource for setting up my code. About half way down there is a video showing the process.<\/p>\n<p>I have spent most of the day trying to get it to run on my machine.<\/p>\n<p>I had a few issues on the setup. Trying to locate the CSV file.<\/p>\n<p>I initially created a directory inside C:\\neo4j\\files\\site.csv, and used the string &#8216;file:\/\/\/C:\/node4j\/files\/site.csv&#8217; in the code to test it.&nbsp; Lost of whirling inside <strong>http:\/\/localhost:7474\/browser\/ <\/strong>but it would not find it.<\/p>\n<p>I put the path in Firefox browser and it found the file, no problem.<\/p>\n<p>In the end I closed the browser down and re-started it and it couldn&#8217;t find the file. It was using a path something like&nbsp; file:\/\/\/<strong>C:\/node4j\/import\/<\/strong>C:\/node4j\/files\/site.csv&nbsp; which messed up the search path, so in the end I put the csv file in the C:\/node4j\/import directory and used the filepath in the code as:<\/p>\n<p>FROM &#8216;<strong>file:\/\/\/C:\/<\/strong>r1.csv&#8217; AS line<\/p>\n<p>which worked. So now Cypher could find the file. I think it has the neo4j\/import&nbsp; in the PATH variable, I may have to check that sometime. Anyway, that worked.<\/p>\n<p>I spent a massive time debugging and so decided that in the Excel File I would filter site, block, each individual room all separately and create a specific CSV file for each of these instances ( you have to start somewhere and I spent so long debugging the import that I thought I would simplify the process (but make it more tedious))<\/p>\n<p>Next I kept on getting error after error in the original code that I had modified, but after constant debugging I have the following code that works:<\/p>\n<ol>\n<li>MATCH (a:site {code:&#8221;DALA190-GRD&#8221;})<\/li>\n<li>LOAD CSV WITH HEADERS<\/li>\n<li>FROM &#8216;file:\/\/\/C:\/site.csv&#8217; AS line<\/li>\n<li>CREATE (e:element {Name:line.Name, Id: toInteger(line.guid)})<\/li>\n<li><strong>set<\/strong><br \/>\ne.type = line.Type,<br \/>\ne.baselife = toInteger(line.Baselife),<br \/>\ne.baselifeLower =toInteger(line.BaselifeLower),<br \/>\ne.baselifeUpper =toInteger(line.BaselifeUpper),<br \/>\ne.c1 = toInteger(line.c1),<br \/>\ne.c2 = toInteger(line.c2),<br \/>\ne.c3 = toInteger(line.c3),<br \/>\ne.c4 = toInteger(line.c4),<br \/>\ne.c5 = toInteger(line.c5),<br \/>\ne.r1 = toInteger(line.r1),<br \/>\ne.r2 = toInteger(line.r2),<br \/>\ne.r3= toInteger(line.r3),<br \/>\ne.r4 = toInteger(line.r4),<br \/>\ne.r5= toInteger(line.r5),<br \/>\ne.unit=line.Unit,<br \/>\ne.unitRate= toFloat(line.Unit_Rate),<br \/>\ne.quantity= toFloat(line.Quantity)<\/li>\n<li>CREATE (e)-[r:BELONGS_TO]-&gt;(a)<\/li>\n<li>RETURN r<\/li>\n<\/ol>\n<p>Going through it:<\/p>\n<p>Item 1. Match identifies the Note that I want to link all the data to. This is the specific site, block or room.<\/p>\n<p>Items 2, 3 is the loading CSV&nbsp; (with headers and pointing to the file<\/p>\n<p>Item 4. This held me up for ages, it creates a node, I think with 2 labels , Name &amp; Id(this being the Guid in the excel sheet). I was trying to get all the data in here, but it would not let me. I then learned that I hat to use the SET command to bring all the other data into the node (going back to <a href=\"https:\/\/www.youtube.com\/watch?v=1kyPUqU-MkE\" target=\"_blank\" rel=\"noopener\"><strong>this Video<\/strong><\/a> in the <strong><a href=\"https:\/\/max-drake.cc\/2018\/05\/31\/python-10-neo4j-graph-database-py2neo\/\" target=\"_blank\" rel=\"noopener\">previous post about neo4j<\/a><\/strong>).<\/p>\n<p>Item 5. Hurray, the set command. I copied the original exercise too well and had [] square brackets in the lines e.baselifeLower =toInteger(line.BaselifeLower<strong>[1]<\/strong>),thinking the [1] was the number of digits in the integers (it wasn&#8217;t it was a reference to a broken up date element and the positions of the elements in the array). [several hours gone on that!!]\n<p>Item 6. I also created the relationship between the new element nodes and the site\/block\/room the elements were to attach to.<\/p>\n<p>Item 7. Just showing the relationships created.<\/p>\n<p>&nbsp;<\/p>\n<p><img decoding=\"async\" class=\"alignnone wp-image-3398 lazyload\" data-src=\"https:\/\/max-drake.cc\/wp-content\/uploads\/2018\/06\/imn11-1024x613.jpg\" alt=\"\" width=\"1496\" height=\"896\" data-srcset=\"https:\/\/max-drake.cc\/wp-content\/uploads\/2018\/06\/imn11-1024x613.jpg 1024w, https:\/\/max-drake.cc\/wp-content\/uploads\/2018\/06\/imn11-300x180.jpg 300w, https:\/\/max-drake.cc\/wp-content\/uploads\/2018\/06\/imn11-768x460.jpg 768w\" data-sizes=\"(max-width: 1496px) 100vw, 1496px\" src=\"data:image\/svg+xml;base64,PHN2ZyB3aWR0aD0iMSIgaGVpZ2h0PSIxIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciPjwvc3ZnPg==\" style=\"--smush-placeholder-width: 1496px; --smush-placeholder-aspect-ratio: 1496\/896;\" \/><\/p>\n<p><img decoding=\"async\" class=\"alignnone wp-image-3395 lazyload\" data-src=\"https:\/\/max-drake.cc\/wp-content\/uploads\/2018\/06\/imn10-1024x620.jpg\" alt=\"\" width=\"1463\" height=\"887\" data-srcset=\"https:\/\/max-drake.cc\/wp-content\/uploads\/2018\/06\/imn10-1024x620.jpg 1024w, https:\/\/max-drake.cc\/wp-content\/uploads\/2018\/06\/imn10-300x182.jpg 300w, https:\/\/max-drake.cc\/wp-content\/uploads\/2018\/06\/imn10-768x465.jpg 768w\" data-sizes=\"(max-width: 1463px) 100vw, 1463px\" src=\"data:image\/svg+xml;base64,PHN2ZyB3aWR0aD0iMSIgaGVpZ2h0PSIxIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciPjwvc3ZnPg==\" style=\"--smush-placeholder-width: 1463px; --smush-placeholder-aspect-ratio: 1463\/887;\" \/><\/p>\n<h3>End comment<\/h3>\n<p>I think using Cypher rather than Python made sense for the learning and debugging. I was debugging neo4j and not python, so learning about the database rather than the relationsahip betweeen python and the database.<\/p>\n<p>I can now test my code out from Python as the next step.<\/p>\n<p>It took me a long time to get this the way I wanted it and I did a lot of deleting of the Element part of the schema.<\/p>\n<p>The next step is to start writing queries on this data to see how it responds.<\/p>\n<p><img decoding=\"async\" class=\"wp-image-3401 aligncenter lazyload\" data-src=\"https:\/\/max-drake.cc\/wp-content\/uploads\/2018\/06\/imn13-1024x670.jpg\" alt=\"\" width=\"1548\" height=\"1014\" data-srcset=\"https:\/\/max-drake.cc\/wp-content\/uploads\/2018\/06\/imn13-1024x670.jpg 1024w, https:\/\/max-drake.cc\/wp-content\/uploads\/2018\/06\/imn13-300x196.jpg 300w, https:\/\/max-drake.cc\/wp-content\/uploads\/2018\/06\/imn13-768x502.jpg 768w\" data-sizes=\"(max-width: 1548px) 100vw, 1548px\" src=\"data:image\/svg+xml;base64,PHN2ZyB3aWR0aD0iMSIgaGVpZ2h0PSIxIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciPjwvc3ZnPg==\" style=\"--smush-placeholder-width: 1548px; --smush-placeholder-aspect-ratio: 1548\/1014;\" \/><\/p>\n<p>One last thing. There are these objects unattached to anything. They are gray, like the &#8220;elements&#8221; but only have an &#8220;id&#8221; from neo4j. I haven&#8217;t been able to get rid of them so far. I may need to find individual &#8220;id&#8221;s and get rid of them that way.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>I had an excel sheet with some property data. I isolated it to one property for this exercise. The property has A Site, A block, A Unit and 8 rooms and 3 cupboards (denoted by room name they were in with a letter &#8216;a&#8217; at the end) In the first part of the process I [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":3398,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[39],"tags":[],"class_list":["post-3355","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-nosql-databases"],"featured_image_src":"https:\/\/max-drake.cc\/wp-content\/uploads\/2018\/06\/imn11.jpg","featured_image_src_square":"https:\/\/max-drake.cc\/wp-content\/uploads\/2018\/06\/imn11.jpg","author_info":{"display_name":"Max Drake","author_link":"https:\/\/max-drake.cc\/?author=1"},"_links":{"self":[{"href":"https:\/\/max-drake.cc\/index.php?rest_route=\/wp\/v2\/posts\/3355","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/max-drake.cc\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/max-drake.cc\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/max-drake.cc\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/max-drake.cc\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=3355"}],"version-history":[{"count":0,"href":"https:\/\/max-drake.cc\/index.php?rest_route=\/wp\/v2\/posts\/3355\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/max-drake.cc\/index.php?rest_route=\/wp\/v2\/media\/3398"}],"wp:attachment":[{"href":"https:\/\/max-drake.cc\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=3355"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/max-drake.cc\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=3355"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/max-drake.cc\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=3355"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}