There may be two reasons why your manner ratings are going down:
1. If you haven't matched for a while, data from your old ratings will continue to be reflected in your account, but with less weight. On the other hand, more recent matches will have more value and weight on your grade.
2. Users from your past matches have pressed the down button. In this case, we recommend you to be a little more gentle and careful while matching.