{"html_url": "https://github.com/simonw/csvs-to-sqlite/releases/tag/0.3", "id": 8556054, "node_id": "MDc6UmVsZWFzZTg1NTYwNTQ=", "tag_name": "0.3", "target_commitish": "master", "name": "csvs-to-sqlite 0.3", "draft": 0, "author": {"value": 9599, "label": "simonw"}, "prerelease": 0, "created_at": "2017-11-17T05:26:07Z", "published_at": "2017-11-17T05:33:39Z", "assets": "[]", "body": "- **Mechanism for converting columns into separate tables**\r\n\r\n  Let's say you have a CSV file that looks like this:\r\n  \r\n      county,precinct,office,district,party,candidate,votes\r\n      Clark,1,President,,REP,John R. Kasich,5\r\n      Clark,2,President,,REP,John R. Kasich,0\r\n      Clark,3,President,,REP,John R. Kasich,7\r\n  \r\n  (Real example from https://github.com/openelections/openelections-data-sd/blob/  master/2016/20160607__sd__primary__clark__precinct.csv )\r\n  \r\n  You can now convert selected columns into separate lookup tables using the new\r\n  --extract-column option (shortname: -c) - for example:\r\n  \r\n      csvs-to-sqlite openelections-data-*/*.csv \\\r\n          -c county:County:name \\\r\n          -c precinct:Precinct:name \\\r\n          -c office -c district -c party -c candidate \\\r\n          openelections.db\r\n  \r\n  The format is as follows:\r\n  \r\n      column_name:optional_table_name:optional_table_value_column_name\r\n  \r\n  If you just specify the column name e.g. `-c office`, the following table will\r\n  be created:\r\n  \r\n      CREATE TABLE \"party\" (\r\n          \"id\" INTEGER PRIMARY KEY,\r\n          \"value\" TEXT\r\n      );\r\n  \r\n  If you specify all three options, e.g. `-c precinct:Precinct:name` the table\r\n  will look like this:\r\n  \r\n      CREATE TABLE \"Precinct\" (\r\n          \"id\" INTEGER PRIMARY KEY,\r\n          \"name\" TEXT\r\n      );\r\n  \r\n  The original tables will be created like this:\r\n  \r\n      CREATE TABLE \"ca__primary__san_francisco__precinct\" (\r\n          \"county\" INTEGER,\r\n          \"precinct\" INTEGER,\r\n          \"office\" INTEGER,\r\n          \"district\" INTEGER,\r\n          \"party\" INTEGER,\r\n          \"candidate\" INTEGER,\r\n          \"votes\" INTEGER,\r\n          FOREIGN KEY (county) REFERENCES County(id),\r\n          FOREIGN KEY (party) REFERENCES party(id),\r\n          FOREIGN KEY (precinct) REFERENCES Precinct(id),\r\n          FOREIGN KEY (office) REFERENCES office(id),\r\n          FOREIGN KEY (candidate) REFERENCES candidate(id)\r\n      );\r\n  \r\n  They will be populated with IDs that reference the new derived tables.\r\n  \r\n  Closes #2\r\n", "repo": {"value": 110509816, "label": "csvs-to-sqlite"}}
{"html_url": "https://github.com/simonw/csvs-to-sqlite/releases/tag/0.5", "id": 8575785, "node_id": "MDc6UmVsZWFzZTg1NzU3ODU=", "tag_name": "0.5", "target_commitish": "master", "name": "csvs-to-sqlite 0.5", "draft": 0, "author": {"value": 9599, "label": "simonw"}, "prerelease": 0, "created_at": "2017-11-19T05:11:27Z", "published_at": "2017-11-19T05:53:25Z", "assets": "[]", "body": "## Now handles columns with integers and nulls in correctly\r\n\r\nPandas does a good job of figuring out which SQLite column types should be\r\nused for a DataFrame - with one exception: due to a limitation of NumPy it\r\ntreats columns containing a mixture of integers and NaN (blank values) as\r\nbeing of type float64, which means they end up as REAL columns in SQLite.\r\n\r\nhttp://pandas.pydata.org/pandas-docs/stable/gotchas.html#support-for-integer-na\r\n\r\nTo fix this, we now check to see if a float64 column actually consists solely\r\nof NaN and integer-valued floats (checked using v.is_integer() in Python). If\r\nthat is the case, we over-ride the column type to be INTEGER instead.\r\n\r\nSee #5 - also a8ab524 and 0997b7b", "repo": {"value": 110509816, "label": "csvs-to-sqlite"}}
{"html_url": "https://github.com/simonw/csvs-to-sqlite/releases/tag/0.6", "id": 8651869, "node_id": "MDc6UmVsZWFzZTg2NTE4Njk=", "tag_name": "0.6", "target_commitish": "master", "name": "csvs-to-sqlite 0.6", "draft": 0, "author": {"value": 9599, "label": "simonw"}, "prerelease": 0, "created_at": "2017-11-24T23:12:10Z", "published_at": "2017-11-24T23:16:45Z", "assets": "[]", "body": "## SQLite full-text search support\r\n\r\n- Added `--fts` option for setting up SQLite full-text search.\r\n\r\n  The `--fts` option will create a corresponding SQLite FTS virtual table, using\r\n  the best available version of the FTS module.\r\n\r\n  https://sqlite.org/fts5.html\r\n  https://www.sqlite.org/fts3.html\r\n\r\n  Usage:\r\n\r\n      csvs-to-sqlite my-csv.csv output.db -f column1 -f column2\r\n\r\n  Example generated with this option: https://sf-trees-search.now.sh/\r\n\r\n  Example search: https://sf-trees-search.now.sh/sf-trees-search-a899b92?sql=select+*+from+Street_Tree_List+where+rowid+in+%28select+rowid+from+Street_Tree_List_fts+where+Street_Tree_List_fts+match+%27grove+london+dpw%27%29%0D%0A\r\n\r\n  Will be used in https://github.com/simonw/datasette/issues/131\r\n- `--fts` and `--extract-column` now cooperate.\r\n\r\n  If you extract a column and then specify that same column in the `--fts` list,\r\n  `csvs-to-sqlite` now uses the original value of that column in the index.\r\n\r\n  Example using CSV from https://data.sfgov.org/City-Infrastructure/Street-Tree-List/tkzw-k3nq\r\n\r\n      csvs-to-sqlite Street_Tree_List.csv trees-fts.db \\\r\n          -c qLegalStatus -c qSpecies -c qSiteInfo \\\r\n          -c PlantType -c qCaretaker -c qCareAssistant \\\r\n          -f qLegalStatus -f qSpecies -f qAddress \\\r\n          -f qSiteInfo -f PlantType -f qCaretaker \\\r\n          -f qCareAssistant -f PermitNotes\r\n\r\n  Closes #9\r\n- Handle column names with spaces in them.\r\n- Added `csvs-to-sqlite --version` option.\r\n\r\n  Using http://click.pocoo.org/5/api/#click.version_option", "repo": {"value": 110509816, "label": "csvs-to-sqlite"}}
{"html_url": "https://github.com/simonw/csvs-to-sqlite/releases/tag/0.6.1", "id": 8652417, "node_id": "MDc6UmVsZWFzZTg2NTI0MTc=", "tag_name": "0.6.1", "target_commitish": "master", "name": "csvs-to-sqlite 0.6.1", "draft": 0, "author": {"value": 9599, "label": "simonw"}, "prerelease": 0, "created_at": "2017-11-25T02:57:01Z", "published_at": "2017-11-25T02:58:25Z", "assets": "[]", "body": "- `-f and -c` now work for single table multiple columns.\r\n\r\n  Fixes #12\r\n", "repo": {"value": 110509816, "label": "csvs-to-sqlite"}}
{"html_url": "https://github.com/simonw/csvs-to-sqlite/releases/tag/0.7", "id": 8656486, "node_id": "MDc6UmVsZWFzZTg2NTY0ODY=", "tag_name": "0.7", "target_commitish": "master", "name": "csvs-to-sqlite 0.7", "draft": 0, "author": {"value": 9599, "label": "simonw"}, "prerelease": 0, "created_at": "2017-11-26T03:11:33Z", "published_at": "2017-11-26T03:14:11Z", "assets": "[]", "body": "- Add -s option to specify input field separator (#13) [Jani Monoses]", "repo": {"value": 110509816, "label": "csvs-to-sqlite"}}
{"html_url": "https://github.com/simonw/csvs-to-sqlite/releases/tag/0.8", "id": 10696701, "node_id": "MDc6UmVsZWFzZTEwNjk2NzAx", "tag_name": "0.8", "target_commitish": "master", "name": "csvs-to-sqlite 0.8", "draft": 0, "author": {"value": 9599, "label": "simonw"}, "prerelease": 0, "created_at": "2018-04-24T15:08:37Z", "published_at": "2018-04-24T15:35:30Z", "assets": "[]", "body": "- `-d` and `-df` options for specifying date/datetime columns, closes #33\r\n- Maintain lookup tables in SQLite, refs #17\r\n- `--index` option to specify which columns to index, closes #24\r\n- Test confirming `--shape` and `--filename-column` and `-c` work together #25\r\n- Use usecols when loading CSV if shape specified\r\n- `--filename-column` is now compatible with `--shape`, closes #10\r\n- `--no-index-fks` option\r\n\r\n  By default, csvs-to-sqlite creates an index for every foreign key column that is\r\n  added using the `--extract-column` option.\r\n\r\n  For large tables, this can dramatically increase the size of the resulting\r\n  database file on disk. The new `--no-index-fks` option allows you to disable\r\n  this feature to save on file size.\r\n\r\n  Refs #24 which will allow you to explicitly list which columns SHOULD have\r\n  an index created.\r\n- Added `--filename-column` option, refs #10\r\n- Fixes for Python 2, refs #25\r\n- Implemented new `--shape` option - refs #25\r\n- `--table` option for specifying table to write to, refs #10\r\n- Updated README to cover `--skip-errors`, refs #20\r\n- Add `--skip-errors` option (#20) [Jani Monoses]\r\n- Less verbosity (#19) [Jani Monoses]\r\n\r\n  Only log `extract_columns` info when that option is passed.\r\n- Add option for field quoting behaviour (#15) [Jani Monoses]", "repo": {"value": 110509816, "label": "csvs-to-sqlite"}}
{"html_url": "https://github.com/simonw/csvs-to-sqlite/releases/tag/0.9", "id": 15022807, "node_id": "MDc6UmVsZWFzZTE1MDIyODA3", "tag_name": "0.9", "target_commitish": "master", "name": "csvs-to-sqlite 0.9", "draft": 0, "author": {"value": 9599, "label": "simonw"}, "prerelease": 0, "created_at": "2019-01-17T05:17:02Z", "published_at": "2019-01-17T05:20:23Z", "assets": "[]", "body": "- Support for loading CSVs directly from URLs, thanks @betatim - #38\r\n- New -pk/--primary-key options, closes #22\r\n- Create FTS index for extracted column values\r\n- Added --no-fulltext-fks option, closes #32\r\n- Now using black for code formatting\r\n- Bumped versions of dependencies", "repo": {"value": 110509816, "label": "csvs-to-sqlite"}}
{"html_url": "https://github.com/simonw/csvs-to-sqlite/releases/tag/0.9.1", "id": 18185234, "node_id": "MDc6UmVsZWFzZTE4MTg1MjM0", "tag_name": "0.9.1", "target_commitish": "master", "name": "csvs-to-sqlite 0.9.1", "draft": 0, "author": {"value": 9599, "label": "simonw"}, "prerelease": 0, "created_at": "2019-06-24T15:16:54Z", "published_at": "2019-06-24T15:21:12Z", "assets": "[]", "body": "* Fixed bug where `-f` option used FTS4 even when FTS5 was available (#41)", "repo": {"value": 110509816, "label": "csvs-to-sqlite"}}
{"html_url": "https://github.com/simonw/csvs-to-sqlite/releases/tag/0.9.2", "id": 18377238, "node_id": "MDc6UmVsZWFzZTE4Mzc3MjM4", "tag_name": "0.9.2", "target_commitish": "master", "name": "csvs-to-sqlite 0.9.2", "draft": 0, "author": {"value": 9599, "label": "simonw"}, "prerelease": 0, "created_at": "2019-07-03T04:36:26Z", "published_at": "2019-07-03T04:37:15Z", "assets": "[]", "body": "Bumped dependencies and pinned pytest to version 4 (5 is incompatible with Python 2.7).", "repo": {"value": 110509816, "label": "csvs-to-sqlite"}}
{"html_url": "https://github.com/simonw/csvs-to-sqlite/releases/tag/1.0", "id": 19056866, "node_id": "MDc6UmVsZWFzZTE5MDU2ODY2", "tag_name": "1.0", "target_commitish": "master", "name": "csvs-to-sqlite 1.0", "draft": 0, "author": {"value": 9599, "label": "simonw"}, "prerelease": 0, "created_at": "2019-08-03T10:50:48Z", "published_at": "2019-08-03T10:58:15Z", "assets": "[]", "body": "This release drops support for Python 2.x #55", "repo": {"value": 110509816, "label": "csvs-to-sqlite"}}