I work in a usability team and in our applications we allways use links to open new pages (navigation), and a buttons to do an action, submit a form, for instance. But now we face a dilema: we have a datatable and we want to edit the rows in a modal window. We want to be consistent at all times, so should we use a link or a button to edit the row in a modal?
Use a button. You're trying to edit something on the current page after which you'll return to the present context. The modal is thus an action even if it temporarily fills the screen like going to a different page.
This are the possibilities that i thought:
1 - use a button "edit" on the last column ( very repetitive, makes the interface more confusing when we have many rows)
2 - use a icone aside the text in the main column to edit, on hover a border appears (very different from what the user expects to see in a table to edit)
3 - use a hyperlink in the main column (the user maybe will expect to edit in another page)
Why have any buttons or links at all?
Can you set the items to be editable directly inline?
If not - if there is more data than just what is shown in each row (ie the data shown on the row is only the most relevant info) then you could use a modal (as you suggested) or expand the row accordian style. These could be performed by clicking anywhere on the row. In the case of the accordian you could also have a small arrow which changes from pointing down to up (or any other visual cue).