Andrew here (https://www.linkedin.com/in/agconti/). I'm the founder of Reactive Streaming, (https://www.reactive-streaming.com), a platform as a service that enables product and growth teams to create interactive live experiences through our easy to use APIs. We make it easy for you to build a live experience by taking care of the foundational live streaming technical challenges so you can focus on what differentiates your product.
- we have the fastest live video in the market, thanks to our propriety live streaming process.
- our interaction services, (a BAAS), lets you create deeply engaging visceral experiences with ease.
I'm located in NYC and I'm looking for a co-founder with deep sales and marketing experience. If you're interested, please reach out to me on LinkedIn!
The tutorial is still under the development and I will update the User model logic for sure (for payments and teams features). The 'wild' overwrite of User model was minimal to set email as required field in User model.
I can deeply vouch for the performance and joy of implementing Full-Text Search with PostgreSQL and Django.
I just finished a a project where we chose Postgres's FTS over using Elastic Search. At the beginning, I was worried about what performance we'd see since we choose to not use ES. But after slight performance tweaking, we had even our least performing queries under 50ms.
You need to look at other stuff than performance - relevancy is probably the biggest thing when implementing search. Is it more relevant than what you experienced with ES?
I'd argue that relevancy is more your application application's design then the underlying system retrieving the results. For example, putting the same dataset in Postgres or ES wouldn't make one deliver more relevant results given equal configurations.
You could lean on the relevancy strategies built in to ES, but in my experience you're better off understanding what relevancy means for your dataset and implementing a strategy yourself. Your millage may vary though, I'd never advocate reimplenting something that's already provided by your chosen tool.
The options and tools for configuring and tweaking relevancy between ES and PostgreSQL's FTS are surprisingly similar for many application use cases. If you're interested you can check out Postgres' search rank and query weighting configurations.
I've had to build complex queries against ElasticSearch and it is specifically designed for things like this. We had custom weightings so when you searched for certain natural keys associated with each item they would rank above everything else, and that is easily do-able with ES. Simultaneously, we would weigh results according to various metadata we had attached to each entry (audio stream languages, subtitles, content owner name, genre, etc.). And finally, if you searched for the name of the media (a movie or an episode in a TV show) the user would see all the matches ranked accordingly, but again weighed according to the content owner and various language features of that media file.
You can probably hack that together with PostgreSQL, but is basically one big query in ES. PG's FTS is still great; but its use-case is slightly different.
That’s cool and I hear you; that’s a complex relevancy definition.
( Maybe surprisingly? ) This is type of query is natively supported by Postgres. That support is robust and mature, you don’t have to hack it together.
ES is a great tool and it’s clear your a fan of it. If you’re interested, I’d recommend you look into Postgres capabilities. It’s not a replacement for ES by any means ( or even a competitor to in my opinion; Postgres isn’t even distributed ). But for specific use cases, you might find that Postgres capabilities surprise you!
(I did leave out some bits that made it more complicated than I indicated.) Also, this was years ago; I know PG has improved its FTS a lot since then. ES is just a useful tool. If you can express your problem in relational terms then a RDBMS is almost always the right choice.
By the way, I am a huge fan of PG and relational databases in general; PG, especially, is a great database, and the first tool I reach for when it comes to data storage. However, we had other requirements (aside from the complexity I left out) to do with versioning and so forth that swung in favour of ES. Ultimately the problem with FTS in RDBMS, for me, boils down to doing FTS across disparate -- let's call them 'documents' -- stored across multiple tables. Basically you have to use materialised views (with manual refreshing) or complex join mechanics that affect performance. Perhaps PG 10 has improved in this area also?
I was about to give him the same answer but you beat me to it. It’s not jus throwing stuff in a field in PG. You can weight multiple bits of data. You can even define multiple different search vectors in separate columns if you want to use different search styles in different situations.
Thanks! I replied to the OP; but weighing, although critical to our needs, was a small part of a larger problem space we had to solve. I hate introducing new technology unless it's strictly required, but we ran into limitations that forced us to down the road of using ES.
> You could lean on the relevancy strategies built in to ES, but in my experience you're better off understanding what relevancy means for your dataset and implementing a strategy yourself.
Ranking is hard. You SHOULD lean on the tools available in Lucene/Solr/ES. PG's ranking tools are a joke in comparison.
> The options and tools for configuring and tweaking relevancy between ES and PostgreSQL's FTS are surprisingly similar for many application use cases.
I don't the sense of the article is that PG FTS is better than ES , but in some situation, as the one I illustrated in my article, you can implement a the same search function with both of them, but with if have PG already in your stack configuring and using it with Django is very simple and convenient.
> For example, putting the same dataset in Postgres or ES wouldn't make one deliver more relevant results given equal configurations.
This is not the case.
The OOTB search capabilities in ElasticSearch (even by default) far, far exceed what you get in PostgreSQL FTS.
Also you're completely contradicting yourself. You say you don't advocate reimplmenting something provided by the tool but then suggest doing exactly that.
But in many situation you don't need all the ES features and you can implement a quite good and fast FTS function directly with your PostgreSQL database if you already use it in your stack.
> The options and tools for configuring and tweaking relevancy between ES and PostgreSQL's FTS are surprisingly similar for many application use cases.
Maybe in very very limited scenarios, but in general, they aren't even close. PostgreSQL doesn't take corpus frequencies into account, which makes it pretty difficult to come anywhere near the relevance ranking quality of Elasticsearch (or any proper search engine).
I second this. You can't let your queries take unusable amounts of time, but below a certain threshold relevancy is infinitely more important.
I'm putting together a product which has a search feature and that uses Django + MySQL and I'm struggling with relevancy. I'd happily accept 500ms queries if that guaranteed me the relevant hit would be on the first page. That's FAR more usable than 50ms queries and then the relevant hit is on page 5.
Full text search in MySQL isn’t in the same ballpark as PG. Thats not a dig at MySQL, just praise for the quality of what you get from the PG implementation.
Second this, even if you are a PG fanboy and a search newbie, you need to pay attention to:
1. issues with i18n and l10n tokenization. Does PG support other languages?
2. At minimum you need to support tf-idf (or something better), it doesn't look like PG supports this either.
3. For extremely dumb ranking, you can have a render/engaged column in PG. For decent production stuff you need a decision tree ranker (or GBDT).
All in all, none of these are there in PG, I'm not familiar with Solr/Lucene either, but please educate yourselves before expressing such strong opinions marketed as the absolute truth.
PG FTS support other languages https://www.postgresql.org/docs/current/static/textsearch-ps...
Anyway the point of my article is not that PG FST is better than ES, but that for a quite good and fast FTS function you can use only Django and PostgreSQL and most of the time you don't need all the other ES features and at the same time your stack will be easier to build and maintain.
I think search relevancy is very important, and I wrote in my article start using PG FTS had permitted to work on search relevancy because I had more time which I used before in ES configuration and maintain another layer in my stack.
Having implemented this for a client in the past I have to agree that it is a cheaper option than ElasticSearch, especially for smaller projects with a lower number of records to index.
ElastiSearch easily gets expensive and the search suggestion is pretty bad.
Thanks for your feedback, I obviously agree with you, but I'm starting to plan to use PG FTS with Django also in some bigger project. I hope to write another article about it in new future.
Depends on the data right. I've seen good performance on larger tables after using GIN indexing with records that rarely needs to be updated and simple queries.
I'm not a expert in PostgreSQL by any means, but reducing cost and learning something never hurts.
- Using a `SearchVectorField` is a must after 500K rows.
- Make keeping this field up to date easy for yourself by populating it using `SeachVector` with a Django pre_save signal or PostgreSQL trigger. This reduces CPU utilization significantly as the parsing and tokenization of the field your searching on is done a head of time.
- Adding a GIN Index on your `SearchVectorField` column will improve performance dramatically.
- You should specify your language configuration for postgres FTS parser. The default parser doesn't do much. It just removes spaces and normalize case. Specifying a language lets the parser make heavier optimizations that noticeably improve performance and the quality of results. If you need support for more then one langue, Django already makes it easy for this configuration to be dynamic.
Not the OP, and I haven't used PG regularly for a few years, but back then I vaguely recollect the query optimizer not behaving consistently based on whether you had a full column with a GIN (or GIST) index and a (potentially partial) index on an expression for some reason. In a nutshell it preferred using the full column with an index rather than the expression index.
Even more importantly in some circumstances, having the full column allows the optimizer to pick another index when it's totally relevant, and filter the relevant rows without needing to recompute the TSV one by one for the subset.
If your "document" is based also on columns on other table, as in the example on my article, you can't have a GIN index on your expression, but you can have a GIN index on your specified column.
The Postgres docs suggest using another column. My guess is that an expression index would be too large if it held the tokenized value of all of your FTS documents. These things can and often are entire written documents. Imagine the index size for 2 Million rows of tokenized documents at 2,000 words each.
You might be able to get away with if if you were indexing a less then large amount of very small documents.
But if you're maintaining a separate ts_vector column, which you then index, you're creating the exact same amount of index data.
Unless you're saying that you would populate the field only on some rows and not all of them, and control this from the app. But you could do that with an expression index, too, assuming the rule is a simple, pure function:
CREATE INDEX index_posts_on_body
ON posts (to_tsvector(body, 'english'))
WHERE published = true;
Cool! I like to see the cookie-cutter stuff. I used pydanny's cookie-cutter back in the day.
However, as I've become more comfortable in the Django and python ecosystem, I have been cutting closer and closer to the stock install.
I think the cookie-cutter stuff is great for when you have a highly-opinionated way to do things (specific test frameworks, databases, patterns, etc).
Some feedback:
Authentication:
I looked at your way of authentication, and I kindly ask you take a look at djoser [1]. They have a nice authentication system for DRF with a password reset flow.
Or also maybe an integration with Auth0?
Storage:
If you replace boto3 with Apache's libcloud [2], then you make this project a little more platform agnostic. I say this as someone who works across multiple platforms. I see that django-storages even plays nicely with it also [3].
And to be honest I'm not so thrilled with Heroku as the deployment target. Maybe just beef up the README with instructions on getting started just on my laptop with pip, cookie-cutter, and docker-compose?
All in all, nice job. I'll add my star to your project and consider it next time the use case arises.
Your recommendations look great, especially libclould.
What do you think can be beefed up in getting started docs? Getting setup on your laptop should be as simple as cookiecutting your app and running `docker-compose up`. Is there something missing that you’d like to see documentation about?
I agree with feedback about Heroku. Professionally, I use Kubernetes and I’m looking forward to when amazon’s managed kubernetes service is out of developer preview so this project can use it. I’ve kept with Heroku because its such a nice platform to get an app started on. It allows you to rapidly develop your app from 0 to 1 without being over encumbered by infrastructure concerns before you need to be. I’ve also kept it in because the integration is just pushing to a docker repository. Just changing the repository destination gives you continuous deployment to anywhere. I like that its flexible and doesn't lock you into a vendor.
Would you rather have continuous deployment to Heroku removed then included?
Regarding the getting started, probably just a a section on the steps to install cookie cutter, then the sample shell commands to bring it up with docker.
And I agree with heroku offering a good value to get started. However, do you think your audience is at that level? Furthermore what happens to my app if/when I graduate to my own server at digital ocean or kube cluster?
Which brings me to another question: who do you think this is for? Beginners or maybe more advanced devs who are needing to prototype ideas or have a greenfield project in front of them?
Myself, I am a capable dev with existing infrastructure at hand (laptop and docker swarm clusters on azure, for example).
Maybe rather than seeing a CI pipeline to heroku you could offer alternatives to push work from local docker to a kube cluster or swarm? I do that with all my projects through bitbucket pipelines CI.
cookiecutter-django-rest now supports Django 1.11+!
cookiecutter-django-rest takes care of the details so you can focus on making your api awesome. Scaffolding a project takes seconds and gives you a solid foundation by baking in Django's best practices. Just add your own resources to the api and start shipping.
We're happy to announce that we just released 0.7.0 this morning! We'd love to hear your feedback.
This is off topic, but is the traffic from HN really that great?
It seems like websites are dead pretty often on HN, but on sites like Reddit where I assume there is a magnitude more traffic they seem to hold up pretty well (with some exceptions).
So what's going on here? Does HN really have a bunch of silent viewers? Or is it something more benign like sites that get submitted are often smaller?
When I posted a recent high ranking blog post[0], I received around 18,000 views over the next 24 hours according to Google Analytics. I don't believe this covers a lot of the bots and associated traffic that also crawled the site.
This amount is nothing for a Jekyll site, I had CPU hovering at around 4%, with a series of services also running on this server. This is, imo, where a "hug of death" should end.
I did however at the time, have several colleagues in web development express shock and awe at the fact the site stayed online with traffic levels such as this. Multiple people tried to school me on the need for Cloudflare at this incredible scale.
A lot of web developers think about "scale" in terms of many bloated CMS platforms, and it is with no surprise that the site in question is running Wordpress.
We had a friend's site a while back where we worked out, with just two desktops and spamming "F5" on the keyboard, we could take the site offline, and take several minutes to recover from after the fact. He went and posted for help on Reddit, and sure enough, the consensus was that his experience involved far more traffic than any website could reasonably be expected to handle.
Your answer therefore lies not in the silent HN users, but in the unusually poor performance of popular CMSs. And although I'm using Wordpress, it's not alone in this situation.
[0] https://news.ycombinator.com/item?id=12973181
See I understand the reasoning behind it, but I don't think that's it alone.
I'm saying that the frequency I see the "HN hug of death" is much more than the "Reddit hug of death" while having many of the same kinds of sites (the "bloated CMS platforms" you talk about).
> This is off topic, but is the traffic from HN really that great?
Hmm I wonder if there are any good traffic statistics. I had a library (msngr.js) hit the front page once over a year or so ago and while it helped me gain a bunch of GitHub stars looking at the traffic itself it wasn't really much at all. In fact it was significantly less than I would have expected. Granted maybe my library just wasn't that interesting to people but I would love to see somewhere with some good HN numbers.
I guess, but I've had a site I own get to the front page of a default sub on reddit, and it's impressive to say the least.
With the multiple-magnitude less comments and votes that HN submissions get, I can't imagine they can get anywhere near that unless HN has a LOT more non-contributing types.
Dynamically generated sites are a Bad Idea when they can be trivially statically generated.
A lot of people still think using WordPress or even Rails for site generation is fine, but both are hard to scale. And if the site is small or academic, it will fall over with not too much traffic.
I wrote a rant [1] about this a couple of weeks ago. I can guarantee that my site will never have a "Database Connection Failed" error (now), because there's no database backing it.
Even better would be if I moved hosting to S3/CloudFront. Dirt cheap, and Amazon isn't going to fall over even if a site hits Reddit's front page. I might not like the bill I get at the end of the month if it really goes crazy, though.
I actually heard of the /. hug of death first, (and your comment feels a little condescending there) and I know of them, but it just feels like HN is such a smaller community compared to Reddit and I see the "hug of death" so much more frequently on HN than I do on reddit (or any other site like them).
I was just curious about the reason why. Like does HN have a large "silent" group, or is it something like that on HN sites climb much faster so I see them when they are down before they've had a chance to recover.
Andrew here (https://www.linkedin.com/in/agconti/). I'm the founder of Reactive Streaming, (https://www.reactive-streaming.com), a platform as a service that enables product and growth teams to create interactive live experiences through our easy to use APIs. We make it easy for you to build a live experience by taking care of the foundational live streaming technical challenges so you can focus on what differentiates your product.
- we have the fastest live video in the market, thanks to our propriety live streaming process.
- our interaction services, (a BAAS), lets you create deeply engaging visceral experiences with ease.
Want the skinny on what we offer? Here is a video: https://www.reactive-streaming.com/product-demo/
I'm located in NYC and I'm looking for a co-founder with deep sales and marketing experience. If you're interested, please reach out to me on LinkedIn!